Information processing system, information processing apparatus, and information processing method

ABSTRACT

An information processing system includes a microphone configured to acquire sound, a display device configured to display information, and at least one controller. The at least one controller is configured to cause the display device to display a first screen at least including one setting item with a set value that can be changed by an instruction, cause the display device to display a second screen including a selection object for transition to the first screen, and acquire text data based on speech acquired through the microphone during display of the second screen and change the set value of the one setting item on the basis of the text data without causing the display device to display the first screen.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation, and claims the benefit, of U.S.patent application Ser. No. 16/922,910, filed on Jul. 7, 2020, whichclaims the benefit of, and priority to, Japanese Patent Application No.2019-131058 filed on Jul. 16, 2019. The above cited patent applicationsare incorporated herein by reference in their entirety.

BACKGROUND Field

The present disclosure relates to an information processing system, aninformation processing apparatus, and an information processing methodthat are capable of enabling operation on displayed information byspeech input. The information processing apparatus can be used as animage processing apparatus, such as a copier or a multifunctionperipheral (MFP), and can also be used as a personal computer (PC) or amobile terminal.

Description of the Related Art

With advances in speech recognition techniques in recent years,apparatuses that execute processing in response to a speech-basedinstruction have been widely used. Japanese Patent Laid-Open No.2019-67258 discloses a system that executes processing for using imagesin response to an instruction provided by user's voice.

Japanese Patent Laid-Open No. 2019-67258 illustrates a method wheremajor instructions, such as execution instructions directed to an imageprocessing apparatus, are provided by voice. However, this document doesnot examine a method where detailed instructions, such as instructionsto change settings, are provided by voice. Such an apparatus has a widerange of setting items, each of which has various screen configurationsand ways of instructions appropriate for the item. To provide a systemhaving high operability, it may be desirable to examine a speech-basedinstruction method appropriate for the configuration of each settingscreen.

SUMMARY

The present disclosure provides an information processing system that iscapable of enabling a speech-based instruction to be made in accordancewith the configuration of a setting screen. In particular, the presentdisclosure provides an information processing system that is capable ofenabling the user to give a speech-based instruction to change a setvalue of a setting item for which a selection object used to change theset value is on a destination screen displayed by transition from thecurrent screen.

Specifically, the present disclosure provides an information processingsystem that includes a microphone configured to acquire sound, a displaydevice configured to display information, and at least one controller.The at least one controller is configured to cause the display device todisplay a first screen at least including one setting item with a setvalue that can be changed by an instruction, cause the display device todisplay a second screen including a selection object for transition tothe first screen, and acquire text data based on speech acquired throughthe microphone during display of the second screen and change the setvalue of the one setting item on the basis of the text data withoutcausing the display device to display the first screen.

Further features will become apparent from the following description ofexemplary embodiments with reference to the attached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a configuration of an information processing system.

FIG. 2 illustrates a configuration of an operation panel.

FIG. 3 is a diagram illustrating a hardware configuration of an audiocontrol apparatus.

FIG. 4 is a diagram illustrating a hardware configuration of a server.

FIG. 5 is a diagram illustrating a hardware configuration of an imageforming apparatus.

FIG. 6 is a diagram illustrating a functional configuration of a controlprogram of the audio control apparatus.

FIG. 7A is a diagram illustrating a functional configuration of acontrol program of the server, and FIG. 7B shows examples of group IDlists.

FIG. 8 is a diagram illustrating a functional configuration of a controlprogram of the image forming apparatus.

FIGS. 9A, 9B, and 9C present a sequence diagram illustratinginteractions between the apparatuses included in the informationprocessing system and also between the control programs of theapparatuses.

FIGS. 10A and 10B show other examples of group ID lists managed and usedby the control program of the server.

FIGS. 11A, 11B, 11C, and 11D show other examples of group ID listsmanaged and used by the control program of the server.

FIGS. 12A, 12B, 12C, and 12D show other examples of group ID listsmanaged and used by the control program of the server.

FIGS. 13A and 13B show other examples of group ID lists managed and usedby the control program of the server.

FIGS. 14A, 14B, and 14C show other examples of group ID lists managedand used by the control program of the server.

FIGS. 15A, 15B, 15C, and 15D show other examples of group ID listsmanaged and used by the control program of the server.

FIGS. 16A, 16B, 16C, and 16D show other examples of group ID listsmanaged and used by the control program of the server.

FIGS. 17A, 17B, and 17C show other examples of group ID lists managedand used by the control program of the server.

FIGS. 18A, 18B, and 18C show other examples of group ID lists managedand used by the control program of the server.

FIGS. 19A, 19B, and 19C show other examples of group ID lists managedand used by the control program of the server.

FIG. 20 shows another exemplary group ID list managed and used by thecontrol program of the server.

FIGS. 21A, 21B, and 21C show other examples of group ID lists managedand used by the control program of the server.

FIG. 22A illustrates a screen displayed by the control program of theimage forming apparatus, FIG. 22B illustrates screen control informationmanaged and used by the control program, FIG. 22C illustrates screencontrol information managed and used by the control program, FIG. 22Dillustrates operation-target determination information managed and usedby the control program, and FIG. 22E illustrates a Japanese version ofthe screen illustrated in FIG. 22A.

FIG. 23A illustrates a screen displayed by the control program of theimage forming apparatus, FIG. 23B illustrates screen control informationmanaged and used by the control program, FIG. 23C illustrates screencontrol information managed and used by the control program, FIG. 23Dillustrates operation-target determination information managed and usedby the control program, and FIG. 23E illustrates a Japanese version ofthe screen illustrated in FIG. 23A.

FIG. 24A illustrates a screen displayed by the control program of theimage forming apparatus, FIG. 24B illustrates screen control informationmanaged and used by the control program, FIG. 24C illustrates screencontrol information managed and used by the control program, FIG. 24Dillustrates operation-target determination information managed and usedby the control program, and FIG. 24E illustrates a Japanese version ofthe screen illustrated in FIG. 24A.

FIG. 25 is a diagram illustrating a processing flow of the controlprogram of the image forming apparatus.

FIG. 26 is a diagram illustrating another processing flow of the controlprogram of the image forming apparatus.

FIGS. 27A, 27B, 27C, and 27D illustrate interactions between theinformation processing system and the user.

FIGS. 28A, 28B, 28C, 28D, and 28E are diagrams illustrating how a screendisplayed by the control program of the image forming apparatus isupdated.

FIG. 29 is a flowchart illustrating conditions under which the controlprogram of the image forming apparatus is applicable.

FIG. 30A illustrates a screen displayed by the control program of theimage forming apparatus, FIG. 30B illustrates screen control informationmanaged and used by the control program, FIG. 30C illustrates screencontrol information managed and used by the control program, FIG. 30Dillustrates a Japanese version of the screen illustrated in FIG. 30A,and FIG. 30E illustrates operation-target determination informationmanaged and used by the control program.

FIG. 31A illustrates a screen displayed by the control program of theimage forming apparatus, FIG. 31B illustrates screen control informationmanaged and used by the control program, FIG. 31C illustrates screencontrol information managed and used by the control program, FIG. 31Dillustrates operation-target determination information managed and usedby the control program, and FIG. 31E illustrates a Japanese version ofthe screen illustrated in FIG. 31A.

FIG. 32A illustrates a screen displayed by the control program of theimage forming apparatus, FIG. 32B illustrates another screen displayedby the control program, FIG. 32C illustrates a Japanese version of thescreen illustrated in FIG. 32A, and FIG. 32D illustrates a Japaneseversion of the screen illustrated in FIG. 32B.

FIG. 33 is a diagram illustrating a processing flow of the controlprogram of the image forming apparatus.

FIGS. 34A, 34B, 34C, 34D, and 34E are diagrams illustrating how a screendisplayed by the control program of the image forming apparatus isupdated.

FIGS. 35A, 35B, 35C, 35D, 35E, 35F, and 35G are diagrams illustratinghow a screen displayed by the control program of the image formingapparatus is updated.

FIG. 36 is a sequence diagram illustrating interactions between theapparatuses included in the information processing system of the presentdisclosure and also between the control programs of the apparatuses.

FIG. 37A illustrates a screen displayed by the control program of theimage forming apparatus, FIG. 37B illustrates screen control informationmanaged and used by the control program, FIG. 37C illustrates screencontrol information managed and used by the control program, FIG. 37Dillustrates operation-target determination information managed and usedby the control program, and FIG. 37E illustrates a Japanese version ofthe screen illustrated in FIG. 37A.

FIG. 38A illustrates a screen displayed by the control program of theimage forming apparatus, FIG. 38B illustrates another screen displayedby the control program, FIG. 38C illustrates a Japanese version of thescreen illustrated in FIG. 38A, and FIG. 38D illustrates a Japaneseversion of the screen illustrated in FIG. 38B.

FIG. 39 is a diagram illustrating a processing flow of the controlprogram of the image forming apparatus.

FIGS. 40A, 40B, 40C, and 40D are diagrams illustrating how a screendisplayed by the control program of the image forming apparatus isupdated.

DESCRIPTION OF THE EMBODIMENTS

Embodiments of the present disclosure will now be described usingspecific configurations, with reference to the drawings. Note thatconfigurations for implementing the present disclosure are not limitedto those described in the embodiments. Some of the configurationsdescribed in the embodiments may be omitted or replaced with equivalentsas long as similar advantageous effects are achievable.

First Embodiment (System Configuration)

FIG. 1 illustrates a configuration of an information processing systemaccording to the present embodiment. As illustrated in FIG. 1, theinformation processing system includes an audio control apparatus 100,an image forming apparatus 101 (image processing apparatus), a server102 (information processing apparatus), a client terminal 103, and agateway 105.

The audio control apparatus 100, the image forming apparatus 101, andthe client terminal 103 are capable of communicating with each other viathe gateway 105 and a network 104 (local area network or LAN). Note thatmore than one audio control apparatus 100, more than one image formingapparatus 101, and more than one client terminal 103 may be configuredto connect each other. The audio control apparatus 100, the imageforming apparatus 101, and the client terminal 103 can communicate withthe server 102 via the gateway 105 and the Internet.

In accordance with an instruction to start a speech operation (speechoperation start instruction) from a user 106, the audio controlapparatus 100 acquires speech of the user 106 and transmits coded audiodata (audio information) to the server 102. The audio control apparatus100 is, for example, a smart speaker. In the present embodiment, theaudio control apparatus 100 and the image forming apparatus 101 areconfigured to be independent of each other. Alternatively, hardwarecomponents (or hardware blocks described below with reference to FIG. 3)and software functions (or software blocks described below withreference to FIG. 6) of the audio control apparatus 100 may be includedin the image forming apparatus 101. That is, the configuration is notlimited to that described above.

Although the image forming apparatus 101 described here is, for example,an MFP having multiple functions, such as copying, scanning, printing,faxing, and e-mailing, the image forming apparatus 101 may be a printeror scanner having a single function. The image forming apparatus 101includes an operation panel 509. The following description assumes thatthe image forming apparatus 101 is a color laser beam MFP.

The server 102 performs speech recognition on audio data representingspeech of the user 106 and acquired by the audio control apparatus 100.From the speech recognition result, the server 102 determines words(word information) that are related to setting operations and jobexecution of the image forming apparatus 101. Additionally, the server102 generates text in accordance with the speech recognition result orthe determined words, and synthesizes audio data for the audio controlapparatus 100 to reproduce speech from the text. Machine learning usinga neural network, such as deep learning, enables the server 102 toprovide accurate results of speech recognition. For example, the server102 performs learning for accurately recognizing speech of a distantuser. Also, the server 102 supports natural language processing. Forexample, through morphological analysis, syntax analysis, semanticanalysis, and context analysis, the server 102 can acquire relevantinformation (words, results of kana-kanji conversion) from naturallanguage received. Such machine learning involves, for example, learninga leaning model on the basis of teacher data containing pairs of audioinformation and text information. Note that the “job” described above isa unit representing a series of image forming operations implemented bythe image forming apparatus 101 (e.g., copying, scanning, or printing)using a print engine 513 or a scanner 515 (see FIG. 5).

The client terminal 103 is, for example, a PC used by the user 106. Theclient terminal 103 issues a print job for printing an electronic fileon the image forming apparatus 101. The electronic file is stored, forexample, in the client terminal 103, any server (not shown) on theInternet, or an external storage device 505 (see FIG. 5) of the imageforming apparatus 101. The client terminal 103 receives image datascanned by the image forming apparatus 101. The operation of the clientterminal 103 will not be described in further detail, as it isirrelevant to the series of descriptions of the present embodiment.

The network 104 enables the audio control apparatus 100, the imageforming apparatus 101, the client terminal 103, and the gateway 105 toconnect with one another. The network 104 enables transmission andreception of various types of data, such as audio data acquired by theaudio control apparatus 100 and transmitted to the server 102, each datatransmitted from the server 102, and print jobs and scan jobs.

The gateway 105 is, for example, a wireless LAN router that is compliantwith the Institute of Electrical and Electronics Engineers (IEEE) 802.11standard series, or may be capable of operating in accordance with adifferent wireless communication system. The gateway 105 may notnecessarily need to be a wireless LAN router, and may be a wired LANrouter compliant with an Ethernet standard, such as 10BASE-T, 100BASE-T,or 1000BASE-T, or may be capable of operating in accordance with adifferent wired communication system. Note that the IEEE 802.11 standardseries described above includes a series of standards belonging to theIEEE 802.11, such as the IEEE 802.11a and IEEE 802.11b.

(Operation Panel of Image Forming Apparatus)

FIG. 2 illustrates a configuration of the operation panel 509 of theimage forming apparatus 101. The operation panel 509 is a display devicethat includes light-emitting diodes (LEDs) and a liquid crystal display(LCD), and displays operations of the user 106 and the internal statesof the apparatus. The operation panel 509 also functions as an acceptingdevice (input device) that accepts operations from the user 106. Theoperation panel 509 includes not only a plurality of hardware keys, butalso a touch panel integral with the LCD. A display unit 200 is an LCDtouch panel where primary mode setting and status display operationstake place.

Keys 201 to 210 are hardware keys, such as tactile switches.

The key 201 represents a numeric keypad (including numeric keys) used toenter numeric values from 0 to 9. The key 202 is an identification (ID)key used for an authentication action (such as a login or logout action)when the apparatus is managed through user/department authentication.

The key 203 is a reset key used to reset the set mode, and the key 204is a guide key used to display a screen showing a description of eachmode. The key 205 is a user mode key used to display a user mode screen.The key 206 is an interrupt key used to perform interrupt copying.

The key 207 is a start key used to start copying, and the key 208 is astop key used to stop a copy job in progress.

The key 209 is a soft power switch. Pressing the key 209 turns off thebacklight of the LCD touch panel 200 and brings the apparatus into alow-power state. The key 210 is an energy saver key. Pressing the key210 brings the apparatus into an energy saving state, and re-pressingthe key 210 enables recovery from the energy saving state.

A key 214 is an adjustment key used to adjust the contrast of the LCDtouch panel 200.

A key 215 is a counter check key. By pressing the key 215, a countscreen showing the total number of copies made so far is displayed onthe LCD touch panel 200.

An LED 216 is an LED indicating that a job is in progress or an image isbeing stored into an image memory. An LED 217 is an error LED indicatingthat the apparatus is in an error state. For example, the LED 217indicates that a paper jam has occurred or a door is open. An LED 218 isa power LED indicating that the main switch of the apparatus is ON.

(Hardware Configuration of Audio Control Apparatus)

FIG. 3 is a diagram illustrating a hardware configuration of acontroller unit 300 of the audio control apparatus 100 and devicesincluded in the audio control apparatus 100.

As illustrated in FIG. 3, the controller unit 300 includes a centralprocessing unit (CPU) 302, a random-access memory (RAM) 303, a read-onlymemory (ROM) 304, an external storage device 305, a network interface(I/F) 306, a microphone I/F 307, an audio controller 309, and a displaycontroller 311, which are connected to a system bus 301 and capable ofcommunicating with one another. The audio control apparatus 100 furtherincludes devices accompanying the controller unit 300. The devicesinclude a microphone 308 serving as a speech input device, a speaker 310serving as an audio output device, and an LED 312 serving as anotification device.

The CPU 302 is a central processing unit that controls the overalloperation of the controller unit 300. The RAM 303 is a volatile memory.The ROM 304 is a nonvolatile memory and stores a boot program for theCPU 302. The external storage device 305 (e.g., secure digital memorycard or SD card) is a storage device with a larger capacity than the RAM303. A control program executed by the controller unit 300 to controlthe audio control apparatus 100 is stored in the external storage device305. The external storage device 305 may be a flash ROM, not the SDcard, or may be replaced with a different type of storage device havinga function equivalent to that of the SD card.

At startup (e.g., when the power is turned on), the CPU 302 executes theboot program stored in the ROM 304. The boot program is for reading thecontrol program stored in the external storage device 305 and developingthe read control program in the RAM 303. After executing the bootprogram, the CPU 302 executes the control program developed in the RAM303 to carry out control. The CPU 302 stores, in the RAM 303, data usedduring execution of the control program, and performs reading from andwriting to the RAM 303. Various settings required during execution ofthe control program can also be stored in the external storage device305, and are read and rewritten by the CPU 302. The CPU 302 communicateswith other devices on the network 104 via the network I/F 306.

The network I/F 306 includes a circuit and an antenna for performingcommunication in accordance with a wireless communication system basedon the IEEE 802.11 standard series. The communication system does notnecessarily need to be a wireless communication system, and may be awired communication system based on the Ethernet standard.

The microphone I/F 307 connects to the microphone 308, converts speechspoken by the user 106 and received by the microphone 308 into codedaudio data, and stores the data in the RAM 303 in accordance with aninstruction from the CPU 302.

The microphone 308 is, for example, a small micro-electro mechanicalsystems (MEMS) microphone mounted on a smartphone, but may be any devicecapable of acquiring speech of the user 106. Three or more microphones308 may be arranged at predetermined positions to calculate thedirection of the incoming speech spoken by the user 106. However, thepresent embodiment can be implemented with one microphone 308. Thenumber of the microphones 308 does not necessarily need to be three ormore.

The audio controller 309 connects to the speaker 310, converts the audiodata into an analog audio signal in accordance with an instruction fromthe CPU 302, and outputs the resulting speech through the speaker 310.

The speaker 310 reproduces an audio response indicating that the audiocontrol apparatus 100 is responding, and also reproduces speechsynthesized by the server 102. The speaker 310 is a sound reproducingdevice for general purposes.

The display controller 311 connects to the LED 312 and controls thedisplay of the LED 312 in accordance with an instruction from the CPU302. In the present embodiment, the display controller 311 mainlycontrols the illumination of the LED 312 for indicating that the audiocontrol apparatus 100 is properly receiving speech input from the user106.

The LED 312 is, for example, a blue LED visible to the user 106. The LED312 is a general-purpose device. The LED 312 may be replaced by adisplay device capable of displaying text and pictures.

(Hardware Configuration of Server)

FIG. 4 is a diagram illustrating a hardware configuration of acontroller unit of the server 102.

As illustrated in FIG. 4, the controller unit includes a CPU 402, a RAM403, a ROM 404, an external storage device 405, and a network I/F 406that are connected to a system bus 401.

The CPU 402 is a central processing unit that controls the overalloperation of the controller unit. The RAM 403 is a volatile memory. TheROM 404 is a nonvolatile memory and stores a boot program for the CPU402. The external storage device 405 (e.g., hard disk drive or HDD) is astorage device with a larger capacity than the RAM 403. A controlprogram executed by the controller unit to control the server 102 isstored in the external storage device 405. The external storage device405 may be a solid state drive (SSD), or may be replaced with adifferent type of storage device having a function equivalent to that ofthe HDD.

At startup (e.g., when the power is turned on), the CPU 402 executes theboot program stored in the ROM 404. The boot program is for reading thecontrol program stored in the external storage device 405 and developingthe read control program in the RAM 403. After executing the bootprogram, the CPU 402 executes the control program developed in the RAM403 to carry out control. The CPU 402 stores, in the RAM 403, data usedduring execution of the control program and performs reading from andwriting to the RAM 403. Various settings required during execution ofthe control program can also be stored in the external storage device405, and are read and rewritten by the CPU 402. The CPU 402 communicateswith other devices on the network 104 via the network I/F 406.

(Hardware Configuration of Image Forming Apparatus)

FIG. 5 is a diagram illustrating a hardware configuration of acontroller unit 500 of the image forming apparatus 101 and devicesincluded in the image forming apparatus 101.

As illustrated in FIG. 5, the controller unit 500 includes a CPU 502, aRAM 503, a ROM 504, an external storage device 505, a network I/F 506, adisplay controller 507, an operation I/F 508, a print controller 512,and a scan controller 514, which are connected to a system bus 501 andcapable of communicating with one another.

The CPU 502 is a central processing unit that controls the overalloperation of the controller unit 500. The RAM 503 is a volatile memory.The ROM 504 is a nonvolatile memory and stores a boot program for theCPU 502. The external storage device 505 (e.g., HDD) is a storage devicewith a larger capacity than the RAM 503. A control program executed bythe CPU 502 to control the image forming apparatus 101 is stored in theexternal storage device 505. The external storage device 505 may be anSSD, or may be replaced with a different type of storage device having afunction equivalent to that of the HDD.

At startup (e.g., when the power is turned on), the CPU 502 executes theboot program stored in the ROM 504. The boot program is for reading thecontrol program stored in the external storage device 505 and developingthe read control program in the RAM 503. After executing the bootprogram, the CPU 502 executes the control program developed in the RAM503 to carry out control. The CPU 502 stores, in the RAM 503, data usedduring execution of the control program and performs reading from andwriting to the RAM 503. Various settings required during execution ofthe control program and image data read by the scanner 515 can also bestored in the external storage device 505, and are read and rewritten bythe CPU 502. The CPU 502 communicates with other devices on the network104 via the network I/F 506.

In accordance with an instruction from the CPU 502, the displaycontroller 507 controls the screen display of the LCD touch panel 200 ofthe operation panel 509 connected to the display controller 507.

The operation I/F 508 allows input and output of operation signals. Theoperation I/F 508 is connected to the operation panel 509. When the LCDtouch panel 200 is pressed, the CPU 502 acquires, via the operation I/F508, coordinates corresponding to the press of the LCD touch panel 200.The operation I/F 508 detects the press of each of the hardware keys 201to 210 on the operation panel 509.

In accordance with an instruction from the CPU 502, the print controller512 transmits a control command or image data to the print engine 513connected to the print controller 512.

The print engine 513 is a printing device that prints the received imagedata (or performs printing) on a sheet in accordance with the controlcommand received from the print controller 512. The print engine 513will not be described further, as its details are beyond the scope here.

In accordance with an instruction from the CPU 502, the scan controller514 transmits a control command to the scanner 515 connected to the scancontroller 514, and writes image data received from the scanner 515 intothe RAM 503.

The scanner 515 is a reading device that reads (or performs reading of)an original on a platen glass (not shown) of the image forming apparatus101 using an optical unit, in accordance with the control commandreceived from the scan controller 514. The scanner 515 will not bedescribed further, as its details are beyond the scope here.

Combining the print engine 513 and the scanner 515 enables photocopyingof the read image of the original.

(Functional Configuration of Audio Control Program of Audio ControlApparatus)

FIG. 6 is a block diagram illustrating a functional configuration of anaudio control program 601 of the audio control apparatus 100, executedby the CPU 302.

The audio control program 601 of the audio control apparatus 100 isstored in the external storage device 305 as described above. Atstartup, the CPU 302 develops the audio control program 601 in the RAM303 and executes it.

A data transmitting and receiving unit 602 transmits and receives datato and from other devices on the network 104 via the network I/F 306 inaccordance with the Transmission Control Protocol/Internet Protocol(TCP/IP). The data transmitting and receiving unit 602 transmits, to theserver 102, audio data representing speech of the user 106 acquired by aspeech acquiring unit 604 (described below). The data transmitting andreceiving unit 602 receives synthesized speech data generated on theserver 102 as a response to the user 106.

A data management unit 603 stores various types of data in apredetermined region of the external storage device 305 and manages thestored data. The various types of data include work data generatedduring execution of the audio control program 601. Specifically, forexample, the data management unit 603 stores and manages volume settingdata for setting the volume of speech reproduced by an audio reproducingunit 605 (described below), authentication information required forcommunicating with the gateway 105, and device information required forcommunicating with the image forming apparatus 101 and the server 102.

The speech acquiring unit 604 converts analog speech of the user 106near the audio control apparatus 100, acquired by the microphone 308,into audio data and temporarily stores the audio data. The speech of theuser 106 is converted to a predetermined format, such as an MPEG-1 AudioLayer-3 (MP3) format, and temporarily stored in the RAM 303 as codedaudio data to be transmitted to the server 102. The start and end timingof processing in the speech acquiring unit 604 is managed by an audiocontrol unit 609 (described below). The audio data may be coded in ageneral-purpose streaming format, and the coded audio data may besequentially transmitted by the data transmitting and receiving unit602.

The audio reproducing unit 605 reproduces, from the speaker 310 via theaudio controller 309, synthesized speech data (audio message) receivedby the data transmitting and receiving unit 602. The timing of audioreproduction in the audio reproducing unit 605 is managed by the audiocontrol unit 609 (described below).

A display unit 606 illuminates the LED 312 via the display controller311. For example, the display unit 606 controls the illumination of theLED 312 when an operation-start detecting unit 607 (described below) hasdetected the start of a speech operation. The timing of when the displayunit 606 is to illuminate the LED 312 is managed by the audio controlunit 609 (described below).

The operation-start detecting unit 607 detects a wake word spoken by theuser 106, or a press of an operation start key (not shown) of the audiocontrol apparatus 100. The operation-start detecting unit 607 thentransmits an operation start notification to the audio control unit 609.Note that the wake word is a predetermined word to be spoken. Theoperation-start detecting unit 607 always detects a wake word fromanalog voice spoken by the user 106 near the audio control apparatus 100and acquired by the microphone 308. The user 106 can operate the imageforming apparatus 101 by saying a wake word and then speaking what he orshe wants to do. Speech processing performed after the operation-startdetecting unit 607 detects a wake word will be described later on below.

A speech-end determining unit 608 determines the end timing ofprocessing in the speech acquiring unit 604. For example, when thespeech of the user 106 stops and a predetermined length of time (e.g.,three seconds) elapses, the speech-end determining unit 608 determinesthat the speech of the user 106 has ended. The speech-end determiningunit 608 then transmits a speech end notification to the audio controlunit 609. The determination of whether the speech has ended may be madeon the basis of a predetermined word spoken by the user 106, not on thebasis of the length of time during which no speech takes place(hereinafter referred to as “blank period”). For example, if apredetermined word, such as “Yes”, “No”, “OK”, “Cancel”, “Finish”,“Start”, or “Begin”, is received, the speech-end determining unit 608may determine that the speech has ended, without waiting for apredetermined length of time. The determination of the speech end may bemade by the server 102, instead of the audio control apparatus 100. Theend of the speech may be determined from the meaning and context of thespeech made by the user 106.

The audio control unit 609 serves as a central control unit that enablesother modules in the audio control program 601 to operate in a mutuallycoordinated manner. Specifically, the audio control unit 609 controlsthe start and end of processing of the speech acquiring unit 604, theaudio reproducing unit 605, and the display unit 606. After the speechacquiring unit 604 acquires audio data, the audio control unit 609performs control such that the data transmitting and receiving unit 602transmits the audio data to the server 102. After the data transmittingand receiving unit 602 receives synthesized speech data from the server102, the audio control unit 609 performs control such that the audioreproducing unit 605 reproduces the synthesized speech data.

The start and end timing of processing in the speech acquiring unit 604,the audio reproducing unit 605, and the display unit 606 will now bedescribed.

Upon receiving an operation start notification from the operation-startdetecting unit 607, the audio control unit 609 starts the processing inthe speech acquiring unit 604. Upon receiving a speech end notificationfrom the speech-end determining unit 608, the audio control unit 609ends the processing in the speech acquiring unit 604. For example,assume that the user 106 speaks a wake word and then says “I want tomake a copy”. In this case, the operation-start detecting unit 607detects the sound of the wake word and transmits an operation startnotification to the audio control unit 609. Upon receiving the operationstart notification, the audio control unit 609 performs control suchthat processing in the speech acquiring unit 604 starts. The speechacquiring unit 604 acquires the subsequent analog speech “I want to makea copy”, converts the acquired analog speech into audio data, andtemporarily stores the audio data. If the speech-end determining unit608 determines that the speech “I want to make a copy” has been followedby a predetermined blank period, the speech-end determining unit 608transmits a speech end notification to the audio control unit 609. Uponreceiving the speech end notification, the audio control unit 609 endsthe processing in the speech acquiring unit 604. Note that the statebetween the start and end of processing in the speech acquiring unit 604will be referred to as a speech processing mode. The display unit 606illuminates the LED 312 to indicate that the audio control apparatus 100is in the speech processing mode.

After the speech-end determining unit 608 determines that the user 106has ended the speech, the audio control unit 609 performs control suchthat the audio data temporarily stored by the speech acquiring unit 604is transmitted from the data transmitting and receiving unit 602 to theserver 102, and then waits for a response from the server 102. Theresponse from the server 102 is, for example, a response messagecomposed of synthesized speech data and a header indicating that this isa response. When the data transmitting and receiving unit 602 receives aresponse message, the audio control unit 609 performs control such thatthe audio reproducing unit 605 reproduces synthesized speech data. Thesynthesized speech data is, for example, “Copy screen will bedisplayed”. Note that the state between the speech end determination andthe end of reproduction of the synthesized speech data will be referredto as a response processing mode. The display unit 606 blinks the LED312 to indicate that the audio control apparatus 100 is in the responseprocessing mode.

After the response processing, as long as the interactive session withthe server 102 continues, the user 106 can speak what he or she wants todo without saying any wake word. The determination of whether theinteractive session has ended is made by the server 102. The server 102transmits an end notification to the audio control apparatus 100 tonotify that the interactive session has ended. Note that the statebetween the end of one interactive session and the start of anotherinteractive session will be referred to as a standby mode. The audiocontrol apparatus 100 is always in standby mode until receiving anoperation start notification from the operation-start detecting unit607. The display unit 606 does not illuminate the LED 312 in the standbymode.

(Functional Configuration of Audio-Data Conversion Control Program ofServer)

FIG. 7A is a block diagram illustrating a functional configuration of anaudio-data conversion control program 701 of the server 102, executed bythe CPU 402. FIG. 7B shows examples of group ID lists used by a group-IDdetermining unit 707 to determine a group ID. In the group ID lists,words having the same meaning or intention in relation to user'soperations on the image forming apparatus 101 are grouped under the sameID. The words listed here are results of speech recognition of wordsspoken to the audio control apparatus 100 by the user 106.

The audio-data conversion control program 701 of the server 102 isstored in the external storage device 405 as described above. Atstartup, the CPU 402 develops the audio-data conversion control program701 in the RAM 403 and executes it.

A data transmitting and receiving unit 702 transmits and receives datato and from other devices on the network 104 via the network I/F 406 inaccordance with the TCP/IP. The data transmitting and receiving unit 702receives audio data representing speech of the user 106 from the audiocontrol apparatus 100. Also, the data transmitting and receiving unit702 transmits the result of group ID determination made by the group-IDdetermining unit 707 and text data generated through speech recognitionprocessing by a speech recognition unit 705 (described below).

A data management unit 703 stores various types of data in apredetermined region of the external storage device 405 and manages thestored data. The various types of data include work data generatedduring execution of the audio-data conversion control program 701, andparameters necessary for an audio data converter 704 to perform speechrecognition processing. Specifically, for example, the data managementunit 703 stores, in a predetermined region of the external storagedevice 405, acoustic and language models for the speech recognition unit705 to convert the audio data received by the data transmitting andreceiving unit 702 into text, and manages the stored models. Also, forexample, the data management unit 703 stores, in a predetermined regionof the external storage device 405, dictionaries for a morphologicalanalysis unit 706 to perform morphological analysis on text, and managesthe stored dictionaries. Also, for example, the data management unit 703stores, in a predetermined region of the external storage device 405,group ID lists for the group-ID determining unit 707 to determine groupIDs, and manages the stored group ID lists. Also, for example, the datamanagement unit 703 stores, in a predetermined region of the externalstorage device 405, a speech database for a speech synthesis unit 708 toperform speech synthesis, and manages the stored speech database. Thedata management unit 703 also stores and manages device informationrequired for communicating with the audio control apparatus 100 and theimage forming apparatus 101.

The audio data converter 704 includes the speech recognition unit 705,the morphological analysis unit 706, the group-ID determining unit 707,and the speech synthesis unit 708. The audio data converter 704 will nowbe described.

The speech recognition unit 705 performs speech recognition processingto convert audio data representing speech of the user 106, received bythe data transmitting and receiving unit 702, into text. The speechrecognition processing involves converting the audio data of speech ofthe user 106 into phonemes using an acoustic model, and then convertingthe phonemes into actual text data using a language model. In thepresent embodiment, the text data is composed of “katakana” textcontaining one or more katakana characters (one type of kana or Japanesephonetic symbols), and “kana/kanji” text obtained by kana-kanjiconversion of the katakana text and containing a mixture of hiraganacharacters (the other type of kana), katakana characters, and kanjicharacters (non-phonetic characters). The kana/kanji text also containsnumbers, alphabetical letters, and signs. A different technique may beused to perform speech recognition processing that converts the audiodata into text data, and the technique is not limited to that describedabove. The speech recognition processing will not be described further,as its details are beyond the scope here.

The morphological analysis unit 706 performs morphological analysis onthe text data obtained through conversion by the speech recognition unit705. The morphological analysis involves extracting morpheme sequencesfrom a dictionary having information about the grammar and parts ofspeech of the language, and determining the part of speech of eachmorpheme. The morphological analysis unit 706 can be implemented byknown software, such as JUMAN, ChaSen, or MeCab, designed formorphological analysis. The morphological analysis unit 706 analyzes thetext data obtained through conversion by the speech recognition unit705, into a morpheme sequence. For example, text data “Kopi-o-shi-tai (Iwant to make a copy)” is analyzed into a morpheme sequence “kopi”, “o”,“shi”, and “tai”, and text data “Esan-kara-eyon-e (From A3 to A4)” isanalyzed into a morpheme sequence “esan”, “kara”, “eyon”, and “e”.

The group-ID determining unit 707 compares the result of morphologicalanalysis made by the morphological analysis unit 706 to the group IDlists shown in FIG. 7B to determine a group ID and generates the resultof group ID determination (hereinafter referred to as “group-IDdetermination result”). For example, the group-ID determining unit 707determines that the morpheme sequence “kopi”, “o”, “shi”, and “tai”matches the group ID “FNC00001” for “kopi (copy)” and generates {ID:FNC00001} as the group-ID determination result. Also, the group-IDdetermining unit 707 determines that the morpheme sequence “esan”,“kara”, “eyon”, and “e” matches two group IDs “PAP00100” and “PAP00101”for “esan (A3)” and “eyon (A4)”, respectively, and generates {ID:PAP00100, ID: PAP00101} as the group-ID determination result.

When the group-ID determination result includes a plurality of IDs, theyare generated in the order they have been subjected to speechrecognition and morphological analysis. For example, when the result ofspeech recognition and morphological analysis is the morpheme sequence“eyon (A4)”, “kara”, “esan (A3)”, and “e”, then {ID: PAP00101, ID:PAP00100} is generated as the group-ID determination result. Acombination of adjacent morphemes may be compared to the group ID listsfor the determination. In this case, if a match is found for onemorpheme in any group ID list and a match is also found for acombination of morphemes including the one morpheme in any group IDlist, then the latter result is used to generate a group-IDdetermination result. For example, if the morpheme sequence is “A” and“4”, then {ID: PAP00101} corresponding to “A4” is generated as thegroup-ID determination result, instead of {ID: CHR00000, ID: NUM00004}corresponding to “A” and “4”. Also, katakana text composed of one ormore katakana characters included in a result of speech recognition andmorphological analysis, and kana/kanji text converted from the katakanatext, may be combined and compared to the group ID lists. For example,the group-ID determining unit 707 first compares the kana/kanji text tothe text in the column labeled “kana/kanji” of the group ID lists. Then,if no corresponding group ID is found, the group-ID determining unit 707compares the katakana text to the text in the column labeled “katakana”of the group ID lists to detect the corresponding group ID. If there isan overlap in “katakana” and a plurality of corresponding group IDs arefound in the group ID lists, a plurality of group-ID determinationresults may be generated as candidates. A group-ID determination resultis thus generated, which accommodates errors in kana-kanji conversionand variations in furigana (or reading aid indicating pronunciation)attached to kanji characters.

The speech synthesis unit 708 performs speech synthesis on the basis ofnotification data received from the image forming apparatus 101. In thespeech synthesis, text prepared in advance to be paired with apredetermined notification is converted to audio data of a predeterminedformat, such as MP3. Exemplary combinations of received notificationdata and text to be synthesized into speech will be described later onbelow with reference to the sequence diagram of FIGS. 9A to 9C. Thespeech synthesis generates, for example, audio data on the basis of thespeech database stored in the data management unit 703. The speechdatabase is, for example, a collection of spoken words with fixedcontents. Although speech synthesis is performed using a speech databasein the present embodiment, a different technique may be used for speechsynthesis. The speech synthesis technique is not limited to that using aspeech database. The speech synthesis will not be described further, asits details are beyond the scope here.

(Functional Configuration of Device Control Program of Image FormingApparatus)

FIG. 8 is a block diagram illustrating a functional configuration of adevice control program 801 of the image forming apparatus 101, executedby the CPU 502.

The device control program 801 of the image forming apparatus 101 isstored in the external storage device 505 as described above. Atstartup, the CPU 502 develops the device control program 801 in the RAM503 and executes it.

A data transmitting and receiving unit 802 transmits and receives datato and from other devices on the network 104 via the network I/F 506 inaccordance with the TCP/IP. The data transmitting and receiving unit 802receives a group-ID determination result generated by the group-IDdetermining unit 707 and text data generated through speech recognitionprocessing by the speech recognition unit 705. The data transmitting andreceiving unit 802 transmits, from the image forming apparatus 101 tothe server 102, a screen update notification indicating that the screendisplay content of the LCD touch panel 200 of the operation panel 509has been updated, and a job state notification indicating the executionstate of the job. The details of the notifications will be describedlater on below with reference to the sequence diagram of FIGS. 9A to 9C.

A data management unit 803 stores various types of data in predeterminedregions of the RAM 503 and the external storage device 505 and managesthe stored data. The various types of data include work data generatedduring execution of the device control program 801, and settingparameters necessary for controlling each device. Specifically, forexample, the data management unit 803 manages job data includingcombinations of setting items and set values of jobs executed by adevice control unit 808 (described below), and also manages machinesetting information defining sheet attributes. Also, the data managementunit 803 stores and manages authentication information required forcommunicating with the gateway 105, and device information required forcommunicating with the server 102. The data management unit 803 alsostores and manages image data used by the image forming apparatus 101 toform images. Also, the data management unit 803 stores screen controlinformation used by a display unit 806 for screen display control, andoperation-target determination information used by an operation-targetdetermining unit 807 to determine an operation target. The datamanagement unit 803 manages the screen control information and theoperation-target determination information for each screen displayed bythe display unit 806.

A scan unit 804 causes the scanner 515 to execute scanning via the scancontroller 514 on the basis of scan job parameter settings for thedevice control unit 808 (described below). The scan unit 804 stores theread image data in the data management unit 803.

A print unit 805 causes the print engine 513 to execute printing via theprint controller 512 on the basis of print job parameter settings forthe device control unit 808 (described below).

The display unit 806 controls the operation panel 509 via the displaycontroller 507. Specifically, the display unit 806 displays useroperable user interface (UI) components (e.g., buttons, pull-down lists,and checkboxes) on the LCD touch panel 200. The display unit 806 updatesthe screen on the basis of the screen control information. Also, thedisplay unit 806 displays, on the LCD touch panel 200, text datarepresenting a speech recognition result received by the datatransmitting and receiving unit 802.

The operation-target determining unit 807 acquires the coordinates of atouched point on the LCD touch panel 200 via the operation I/F 508 anddetermines, as an operation target, a UI component that is displayed onthe LCD touch panel 200 and operable by the user 106. Also, when a pressof one of the hardware keys 201 to 210 on the operation panel 509 isdetected, the operation-target determining unit 807 determines the oneof the hardware keys 201 to 210 as the UI component to be operated(operation target). Also, the operation-target determining unit 807determines the operation target on the basis of the group-IDdetermination result received by the data transmitting and receivingunit 802. The operation target is, for example, any of the UI componentsincluded in the screen of the LCD touch panel 200 and operable by theuser 106, or any of the hardware keys 201 to 210 included in theoperation panel 509.

Additionally, the operation-target determining unit 807 reads screencontrol information corresponding to the UI component determined as theoperation target and determines, on the basis of the read screen controlinformation, processing to be performed upon accepting the operation.For example, the operation-target determining unit 807 instructs thedisplay unit 806 to update the displayed content on the screen, ortransmits parameters for a job specified by a user's operation and aninstruction to start the job to the device control unit 808. Theprocessing performed by the operation-target determining unit 807 willbe described in detail later on below with reference to FIG. 26.

The device control unit 808 instructs the print controller 512 and thescan controller 514 to control the print engine 513 and the scanner 515,respectively. For example, when the display unit 806 detects a press ofthe start key 207 during display of a copy function screen, the devicecontrol unit 808 receives parameters for a copy job and a job startinstruction from the operation-target determining unit 807. Then, on thebasis of the job parameters, the device control unit 808 performscontrol such that image data read by the scanner 515 is printed on asheet by the print engine 513. The mechanism of scan and print controlwill not be described further, as it is beyond the scope here.

(System Control Sequence)

FIGS. 9A to 9C present a sequence diagram illustrating interactionsbetween the apparatuses included in the information processing systemillustrated in FIG. 1 and also between the control programs of theapparatuses illustrated in FIG. 6, FIG. 7A, and FIG. 8. FIGS. 9A to 9Cillustrate a sequence in which, after the audio control apparatus 100receives a speech operation based on voice of the user 106, the imageforming apparatus 101 executes the corresponding processing and returnsan audio response representing the result of the execution to the user106. This sequence example illustrated in FIGS. 9A to 9C assumes thatthe audio control apparatus 100, the image forming apparatus 101, andthe server 102 are ready to communicate with each other. This sequenceexample also assumes that after startup (e.g., after the power is turnedon), the image forming apparatus 101 displays a main menu screen whichallows the user 106 to call a function, such as copying, scanning, orprinting.

First, in step S905 (hereinafter referred to as S905 and so on), theuser 106 instructs the audio control apparatus 100 to start a speechoperation. The instruction to start the speech operation is sent whenthe user 106 says a wake word or presses the operation start key of theaudio control apparatus 100. The instruction to start the speechoperation is detected by the operation-start detecting unit 607.

When the instruction to start the speech operation is detected, thedisplay unit 606 of the audio control program 601 illuminates the LED312 in S906 to indicate that the audio control apparatus 100 is in thespeech processing mode. At the same time, the speech acquiring unit 604starts to perform processing.

In S907, the user 106 instructs the audio control apparatus 100 to calla function. This function calling instruction is sent when the user 106speaks, for example, “I want to make a copy” or “Display the copyscreen” after saying a wake word in S905. From the speech acquired bythe speech acquiring unit 604, audio data is generated. After the elapseof a predetermined blank period, the speech-end determining unit 608determines that the speech has ended.

In response to the speech-end determination, the display unit 606 of theaudio control program 601 blinks the LED 312 in S908 to indicate thatthe audio control apparatus 100 is in the response processing mode. Atthe same time, the speech acquiring unit 604 ends its processing.

After the audio data representing the function calling instructionreceived in S907 is generated, the data transmitting and receiving unit602 transmits the generated audio data to the server 102 in S909.

In S910, the speech recognition unit 705 of the audio-data conversioncontrol program 701 performs speech recognition on the audio datareceived by the data transmitting and receiving unit 702. By the speechrecognition, for example, text is generated from the speech“Kopi-shi-tai (I want to make a copy)” spoken by the user 106.

In S911, the morphological analysis unit 706 of the audio-dataconversion control program 701 performs morphological analysis on thetext generated in S910. By morphological analysis, for example, the text“Kopi-shi-tai” is analyzed into a morpheme sequence “kopi”, “shi”, and“tai”.

In S912, the group-ID determining unit 707 of the audio-data conversioncontrol program 701 performs group ID determination on the text analyzedinto a morpheme sequence. By the group ID determination, for example,the morpheme sequence “kopi”, “shi”, and “tai” is compared to the groupID lists shown in FIG. 7B, and {ID: FNC00001} is generated as thegroup-ID determination result.

In S913, the data transmitting and receiving unit 702 of the audio-dataconversion control program 701 transmits the group-ID determinationresult obtained in S912 to the image forming apparatus 101.

In S914, the operation-target determining unit 807 of the device controlprogram 801 performs operation target determination on the group-IDdetermination result received by the data transmitting and receivingunit 802. For example, from the group-ID determination result {ID:FNC00001}, the operation-target determining unit 807 determines that the“Copy” button in the main menu screen has been selected as the operationtarget.

In S915, in accordance with the determination made in S914, the displayunit 806 updates the displayed content of the screen. For example, onthe basis of the processing described above, if it is determined, fromthe speech “Kopi-shi-tai” spoken by the user 106, that the target of theoperation is the “Copy” button displayed on the screen, the display unit806 displays the copy function screen in the same manner as when the“Copy” button is touched on the LCD touch panel 200.

In S916, the data transmitting and receiving unit 802 transmits, to theserver 102, a screen update notification indicating that displayedcontent on the screen has been updated. For example, when the displayunit 806 switches the display from the main menu screen to the copyfunction screen, the data transmitting and receiving unit 802 transmitstext data “copy function screen displayed” as a screen updatenotification.

In S917, the speech synthesis unit 708 of the audio-data conversioncontrol program 701 performs speech synthesis on the screen updatenotification received by the data transmitting and receiving unit 702.Specifically, the speech synthesis unit 708 synthesizes speech frompredetermined text data corresponding to the screen update notification.For example, if the screen update notification is “copy function screendisplayed”, the speech synthesis unit 708 synthesizes the speech “Copyscreen has been displayed” from the corresponding text data.

In S918, the audio data (synthesized speech data) generated throughspeech synthesis in S917 by the speech synthesis unit 708 is transmittedby the data transmitting and receiving unit 702 to the audio controlapparatus 100. The synthesized speech data is received by the datatransmitting and receiving unit 602 of the audio control program 601.

In S919, the audio reproducing unit 605 reproduces the synthesizedspeech data received in S918. For example, the synthesized speech data“Copy screen has been displayed” generated in S917 is reproduced throughthe speaker 310.

In S920, after the synthesized speech data is reproduced, the displayunit 606 of the audio control program 601 illuminates the LED 312 againto indicate that the audio control apparatus 100 is in the speechprocessing mode. At the same time, the speech acquiring unit 604 startsprocessing again.

In S921, the user 106 instructs the audio control apparatus 100 to calla setting screen. The instruction to call a setting screen is sent whenthe user 106 speaks, for example, “Yoshi-o-sentaku (Select paper)”. Fromthe speech acquired by the speech acquiring unit 604, audio data isgenerated. When a predetermined blank period elapses after the speech ofthe user 106, the speech-end determining unit 608 determines that thespeech has ended.

In S922, the same operation as that in S908 is performed.

In S923, the speech operation processing similar to that performed inS909 to S918 is performed. The difference is that in S923, the displayunit 806 updates the screen in accordance with the instruction (sent in5921) to call a setting screen. For example, when the group-IDdetermination result {ID: PAP00000, ID: OPR00040} is obtained duringdisplay of the copy function screen, the display unit 806 displays apaper selection screen.

In S924, the audio reproducing unit 605 reproduces the synthesizedspeech data generated by speech synthesis in S923. For example, when thepaper selection screen is displayed as a setting screen in S923, theaudio reproducing unit 605 reproduces the synthesized speech data “Paperselection screen has been displayed. Please select the paper” throughthe speaker 310.

In S925, the same operation as that in S920 is performed.

In S926, the user 106 instructs the audio control apparatus 100 tochange the setting. The setting change instruction is sent when the user106 speaks, for example, “Eyon (A4)”. When a predetermined blank periodelapses after the speech of the user 106, the speech-end determiningunit 608 determines that the speech has ended.

In S927, the same operation as that in S908 is performed.

In S928, the speech operation processing similar to that performed inS909 to S918 is performed. The difference is that in S928, the displayunit 806 changes a set value displayed on the setting screen (i.e., thedisplay unit 806 executes processing that accepts an instruction tochange a set value) in accordance with the instruction (sent in S926) tochange the setting. For example, when the group-ID determination result{ID: PAP00101} is obtained during display of the paper selection screen,the display unit 806 updates the paper selection screen to change theset value for the type of paper to A4.

In S929, the audio reproducing unit 605 reproduces the synthesizedspeech data generated by speech synthesis in S928. For example, when theset value for the type of paper is changed and displayed in S928, theaudio reproducing unit 605 reproduces the synthesized speech data “A4paper has been selected” through the speaker 310.

In S930, the same operation as that in S920 is performed.

In S931, the user 106 instructs the audio control apparatus 100 toconfirm the setting. The instruction to confirm the setting is sent whenthe user 106 speaks, for example, “Yoshi-no-sentaku-o-kanryo (Papersection has been completed)”. When a predetermined blank period elapsesafter the speech of the user 106, the speech-end determining unit 608determines that the speech has ended.

In S932, the same operation as that in S908 is performed.

In S933, the speech operation processing similar to that performed inS909 to S918 is performed. The difference is that in S933, the displayunit 806 updates the displayed content on the LCD touch panel 200 toenable transition to the original screen, in accordance with theinstruction to confirm the setting in 5931. For example, when thegroup-ID determination result {ID: PAP00000, ID: OPR00040, ID: OPR00012}is obtained during display of the paper selection screen, the displayunit 806 displays the copy function screen displayed at the point of5921.

In S934, the audio reproducing unit 605 reproduces the synthesizedspeech data generated by speech synthesis in S928. For example, thesynthesized speech data “You can start copying” is reproduced throughthe speaker 310.

In S935, the same operation as that in S920 is performed.

In S936, the user 106 instructs the audio control apparatus 100 toexecute a job. The job execution instruction is sent when the user 106speaks, for example, “Kopi-sutato (Start copying)”. When a predeterminedblank period elapses after the speech of the user 106, the speech-enddetermining unit 608 determines that the speech has ended.

The processing in S937 to S942 is similar to that performed in S908 toS913.

In S943, the operation-target determining unit 807 of the device controlprogram 801 performs operation target determination on the group-IDdetermination result received by the data transmitting and receivingunit 802. If the group-ID determination result is {ID: FNC00001, ID:OPR00011}, the operation-target determining unit 807 determines that the“Copy Start” button displayed on the screen or the start key 207, whichis a hardware key, has been operated.

In S944, the display unit 806 displays the job execution screen inaccordance with the determination made in S943. For example, if theoperation-target determining unit 807 determines, on the basis of theprocessing performed so far, that the speech “Kopi-sutato (Startcopying)” spoken by the user 106 is an operation on the start key 207,the display unit 806 displays a copy job start screen.

In S945, a job is executed in accordance with job parameters set on thescreen of the image forming apparatus 101.

In S946, the data transmitting and receiving unit 802 transmits a jobstate notification to the server 102. Specifically, the datatransmitting and receiving unit 802 transmits information (job startnotification) indicating that job execution has started. For example,when a copy job has started, the text data “copy job started” istransmitted as a job state notification to the server 102.

In S947, the data transmitting and receiving unit 702 of the audio-dataconversion control program 701 receives the job state notification, andthe speech synthesis unit 708 synthesizes speech from predetermined textdata corresponding to the job state notification (job startnotification) received. For example, if the job state notification is“copy job started”, the speech synthesis unit 708 synthesizes the speech“Copying will start” from the corresponding text data.

In S948, the same operation as that in S918 is performed.

In S949, the audio reproducing unit 605 reproduces the synthesizedspeech data received in S948. For example, the synthesized speech data“Copying will start” generated in S947 is reproduced through the speaker310.

In S950, the data transmitting and receiving unit 802 transmits a jobstate notification to the server 102. Specifically, the datatransmitting and receiving unit 802 transmits information (job endnotification) indicating that job execution has ended. For example, whenthe copy job has ended, the text data “copy job completed” istransmitted as a job state notification to the server 102.

In S951, the data transmitting and receiving unit 702 of the audio-dataconversion control program 701 receives the job state notification, andthe speech synthesis unit 708 synthesizes speech from predetermined textdata corresponding to the job state notification (job end notification)received. For example, if the job state notification is “copy jobcompleted”, the speech synthesis unit 708 synthesizes the speech“Copying has been completed” from the corresponding text data.

In S952, in response to completion of the job executed in S945, thedisplay unit 806 displays a job end screen. For example, when executionof a copy job ends, the display unit 806 displays a copy job end screen.

In S953, the data transmitting and receiving unit 602 receives, from theserver 102, the synthesized speech data generated in S951. The datatransmitting and receiving unit 602 also receives, from the server 102,a session end notification notifying the audio control apparatus 100that the interactive session with the user 106 is to be completed.

In S954, the audio reproducing unit 605 reproduces the synthesizedspeech data received in S953. For example, the synthesized speech data“Copying has been completed” generated in S951 is reproduced through thespeaker 310.

In S955, in response to receiving the session end notification in S953,the display unit 606 of the audio control program 601 turns off the LED312 to indicate that the audio control apparatus 100 is in standby mode.

In S956, in response to receiving the session end notification in S953,the audio control apparatus 100 is brought into standby mode.

Even when the sequence diagram shows that the LED 312 is blinking toindicate that the audio control apparatus 100 is in the responseprocessing mode, the audio control apparatus 100 is ready to receiveinput of a wake word. Therefore, after speaking a wake word, the user106 may say “Cancel” or “Stop” to forcibly terminate the interactivesession.

(Screen Control Information and Operation-Target DeterminationInformation)

FIGS. 10A and 10B, 11A to 11D, 12A to 12D, 13A and 13B, 14A to 14C, 15Ato 15D, 16A to 16D, 17A to 17C, 18A to 18C, 19A to 19C, 20, and 21A to21C present other examples of the group ID lists shown in FIG. 7B. FIG.22A illustrates a screen displayed by the display unit 806, and FIG. 22Eillustrates a Japanese version of the screen illustrated in FIG. 22A.FIG. 23A illustrates a screen displayed by the display unit 806, and 23Eillustrates a Japanese version of the screen illustrated in FIG. 23A.FIG. 24A illustrates a screen displayed by the display unit 806, andFIG. 24E illustrates a Japanese version of the screen illustrated inFIG. 24A.

The screen control information is generally divided into two types: theinformation such as that shown in FIG. 22B, FIG. 23B, and FIG. 24B, andthe information such as that shown in FIG. 22C, FIG. 23C, and FIG. 24C.The data management unit 803 manages the screen control information foreach screen.

The screen control information of the former type includes UI componentseach included in the screen and determined as an operation target by apress of the LCD touch panel 200 or a press of the hardware keys 201 to210, and screen control and internal processing executed when thecorresponding UI component is determined as an operation target. The UIcomponents each listed as “UI component (touch and speech operable)” inthe screen control information is also determined as an operation targetby speech of the user 106 through operation target determinationperformed by the operation-target determining unit 807.

The screen control information of the latter type includes UI componentseach included in the screen and determined as an operation target onlyby speech of the user 106 through operation target determination, andscreen control and internal processing executed when the correspondingUI component is determined as an operation target.

FIGS. 22B and 22C, FIGS. 23B and 23C, and FIGS. 24B and 24C show textthat describes “screen control during operation” and “internalprocessing during operation” in the screen control information. However,the data management unit 803 manages such information in a data formatthat can be identified by the display unit 806 and the operation-targetdetermining unit 807 that use the information.

In the operation-target determination information shown in FIG. 22D,FIG. 23D, and FIG. 24D, the UI components of the screen are eachassociated in advance with at least one group-ID set. Each of the groupID sets included in the operation-target determination information mayeither be a group ID in the group ID lists shown in FIGS. 10A and 10B,11A to 11D, 12A to 12D, 13A and 13B, 14A to 14C, 15A to 15D, 16A to 16D,17A to 17C, 18A to 18C, 19A to 19C, 20, and 21A to 21C, or a combinationof such group IDs. For example, the group ID set “COL00110+COL00000”matches the group-ID determination result {ID: COL00110, ID: COL00000}.The data management unit 803 manages the operation-target determinationinformation for each screen such that there is no overlap in group IDset within one piece of operation-target determination information. The“example of user speech during operation” shown in FIG. 22D, FIG. 23D,and FIG. 24D is merely reference information for describing the presentembodiment, and does not necessarily need to be included in theoperation-target determination information managed by the datamanagement unit 803.

FIG. 22D shows exemplary speech operations performed by the user 106 whoviews the screen illustrated in FIG. 22E. FIG. 23D shows exemplaryspeech operations performed by the user 106 who views the screenillustrated in FIG. 23E. FIG. 24D shows exemplary speech operationsperformed by the user 106 who views the screen illustrated in FIG. 24E.

In the operation-target determination information shown in FIG. 22D,FIG. 23D, and FIG. 24D, the UI components of the screen are eachassociated in advance with a coordinate range for accepting a touchoperation on the LCD touch panel 200. The “touch coordinate range” inthe operation-target determination information shows a range foraccepting a touch operation on the corresponding UI component, andbasically matches the display position and range of the UI component onthe screen. The “touch coordinate range” in the operation-targetdetermination information is represented by rectangle information, whichincludes X and Y coordinate values indicating the upper-left position ofthe rectangle and the width (W) and height (H) values of the rectangle.

FIG. 22A illustrates a color selection screen 2200 displayed by thedisplay unit 806 for the user 106 to select a print color. The colorselection screen 2200 includes UI components 2201, 2202, 2203, 2204,2205, 2211, 2212, and 2213 that can be operated by the user 106.

The screen control information in FIG. 22B shows that when, for example,the button 2201 labeled “Auto (Color/Black)” is determined as theoperation target, the display unit 806 performs control so as tohighlight the button 2201. Also, when the button 2204 labeled “SingleColor” is determined as the operation target, the display unit 806performs control so as to enable transition to a single-color advancedsettings screen (not shown). Also, when the hardware key (start key) 207is determined as the operation target, the device control unit 808starts execution of a copy job and the display unit 806 performs controlso as to enable transition to a copy job start screen (not shown).

The screen control information in FIG. 22C is empty. This indicates thatthe color selection screen 2200 does not have a UI component that can beoperated only by speech.

The operation-target determination information in FIG. 22D shows thatwhen the group-ID determination result is {ID: DRG00008} or {ID:DRG00008, ID: COL00000, ID: COL00102}, the button 2201 is determined asthe operation target. For example, assume that the user 106 speaks “Jido(Auto)” during display of the color selection screen 2200. In this case,from the group-ID determination result {ID: DRG00008} generated by thegroup-ID determining unit 707, the operation-target determining unit 807determines that the button 2201 is the operation target.

The operation-target determination information in FIG. 22D also showsthat the buttons 2201, 2202, 2203, 2204, 2205, 2211, 2212, and 2213,each having rectangle information representing the touch coordinaterange, are capable of accepting a touch operation on the LCD touch panel200. For example, when touch coordinates are (X: 200, Y: 250), theoperation-target determining unit 807 determines the button 2201included in the touch coordinate range in the operation-targetdetermination information as the operation target. The hardware keys 203and 207, for which the corresponding cells of the touch coordinateranges in FIG. 22D are each provided with a diagonal line, are UIcomponents that are each determined as the operation target on the basisof the detection of a press of the hardware key included in theoperation panel 509.

FIG. 23A illustrates a two-sided copy settings screen 2300 displayed bythe display unit 806 for the user 106 to select the type of two-sidedcopying. The two-sided copy settings screen 2300 includes UI components2301, 2302, 2303, 2304, 2311, 2321, 2331, 2332, and 2333 that can beoperated by the user 106.

The screen control information in FIG. 23B shows that when, for example,the button 2321 labeled “Set Details” is determined as the operationtarget, the display unit 806 performs control so as to enable transitionto a flipping-format advanced settings screen 2400 illustrated in FIG.24A.

The screen control information in FIG. 23C shows information about a UIcomponent that can be operated only by speech of the user 106. Forexample, unlike the UI components (selection objects) included in thescreen control information in FIG. 23B, the display label 2311 shown inthe screen control information in FIG. 23C cannot be operated by touchof the user 106. The display label 2311 is a UI component for displayinga set value that is set on the flipping-format advanced settings screen2400 illustrated in FIG. 24A.

The operation-target determination information in FIG. 23D shows thatwhen the group-ID determination result is {ID: OPR00048, ID: OPR00040}or {ID: LYT00023, ID: OPR00040}, the button 2321 is determined as theoperation target. For example, assume that the user 106 speaks“Shosai-settei (Set details)” during display of the two-sided copysettings screen 2300. In this case, from the group-ID determinationresult {ID: OPR00048, ID: OPR00040} generated by the group-IDdetermining unit 707, the operation-target determining unit 807determines that the button 2321 is the operation target. Then, on thebasis of this determination result and the screen control information,the flipping-format advanced settings screen 2400 illustrated in FIG.24A is displayed.

The operation-target determination information in FIG. 23D shows thatwhen the group-ID determination result is {ID: LYT00023, ID: POS00023,ID: LYT00023} or {ID: LYT00023, ID: POS00022, ID: LYT00023}, the displaylabel 2311 is determined as the operation target. For example, assumethat the user 106 speaks “Hiraki-kata-o-joge-biraki-ni-henko (Change theflipping format to calendar type (up and down))” during display of thetwo-sided copy settings screen 2300. In this case, from the group-IDdetermination result {ID: LYT00023, ID: POS00022, ID: LYT00023}generated by the group-ID determining unit 707, the operation-targetdetermining unit 807 determines that the display label 2311 is theoperation target. Then, on the basis of this determination result andthe screen control information, the displayed content of the displaylabel 2311 is updated. That is, the data management unit 803 manages thegroup ID sets in the operation-target determination information suchthat they include group IDs corresponding to UI components of thedestination screen subsequently displayed by transition.

The operation-target determination information in FIG. 23D also showsthat the buttons 2301, 2302, 2303, 2304, 2321, 2331, 2332, and 2333,each having rectangle information representing the touch coordinaterange, are capable of accepting a touch operation on the LCD touch panel200. The hardware keys 203 and 207, for which the corresponding cells ofthe touch coordinate ranges in FIG. 23D are each provided with adiagonal line, are UI components that are each determined as theoperation target on the basis of the detection of a press of thehardware key included in the operation panel 509. In FIG. 23D, the touchcoordinate range of the display label 2311 is indicated by “-”. Thismeans that the display label 2311 appears on the LCD touch panel 200,but is not designed to accept a touch operation.

FIG. 24A illustrates the flipping-format advanced settings screen 2400displayed by the display unit 806 for the user 106 to select theflipping format of two-sided copies. The flipping-format advancedsettings screen 2400 includes UI components 2401, 2402, 2411, and 2421that can be operated by the user 106. FIGS. 24B and 24C show screencontrol information of the flipping-format advanced settings screen2400, and FIG. 24D shows operation-target determination information ofthe flipping-format advanced settings screen 2400. (Processing Flow ofDevice Control Program of Image Forming Apparatus)

FIG. 25 is a flowchart illustrating processing performed by the devicecontrol program 801 of the image forming apparatus 101 for acceptinginput operation. The processing for accepting input operation isexecuted after the data transmitting and receiving unit 802 receives agroup-ID determination result. Alternatively, this processing isexecuted after the operation I/F 508 detects a touch operation on theLCD touch panel 200 or a press of any of the hardware keys 201 to 210 onthe operation panel 509. For example, FIG. 25 illustrates the processingflow of the device control program 801 corresponding to S914, S915,S923, S928, S933, S943 to S945, and S952 in the sequence diagram ofFIGS. 9A to 9C. FIG. 26 is a flowchart illustrating details of operationtarget determination in S2502, which is part of the processing of thedevice control program 801 illustrated in FIG. 25.

Hereinafter, a processing flow performed while the display unit 806 isdisplaying the two-sided copy settings screen 2300 in FIG. 23A will bedescribed as an example.

In S2501, screen control information and operation-target determinationinformation that correspond to the screen displayed by the display unit806 are acquired.

When the display unit 806 displays the two-sided copy settings screen2300 illustrated in FIG. 23A, the screen control information in FIGS.23B and 23C and the operation-target determination information in FIG.23D corresponding to the two-sided copy settings screen 2300 areacquired.

In S2502, the operation-target determining unit 807 performs operationtarget determination using the operation-target determinationinformation acquired in S2501. The operation target determinationinvolves determining, as an operation target, a UI component of thescreen displayed on the LCD touch panel 200 or any of the hardware keys201 to 210 included in the operation panel 509.

The details of S2502 will now be described with reference to FIG. 26.

First, in S2601, the type of input operation is determined. In theflowchart of FIG. 25, if a group-ID determination result is received bythe data transmitting and receiving unit 802, the operation-targetdetermining unit 807 determines that a speech operation has beenexecuted. Alternatively, if the operation I/F 508 detects a touchoperation on the LCD touch panel 200 or a press of any of the hardwarekeys 201 to 210 on the operation panel 509, the operation-targetdetermining unit 807 determines that a touch or press operation has beenexecuted. If the type of input operation is speech, that is, if, as inFIG. 9A, the user 106 speaks after instructing to start a speechoperation in S905 and the data transmitting and receiving unit 802receives a group-ID determination result from the server 102, theprocess proceeds to S2602. If the type of input operation is a touchoperation or a press of a hardware key, the process proceeds to S2610.

In S2602 and S2603, the operation-target determining unit 807 determineswhether the group-ID determination result received by the datatransmitting and receiving unit 802 matches any of the group ID sets inthe operation-target determination information acquired in S2501. Forexample, if the user 106 speaks “Furukara (Full color)” irrelevant tothe two-sided copy settings screen 2300 being displayed, theoperation-target determining unit 807 determines that the group-IDdetermination result {ID: COL00201} determined by the group-IDdetermining unit 707 does not match any of the group ID sets in theoperation-target determination information in FIG. 23D, and the processproceeds to S2609.

In S2609, the operation-target determining unit 807 determines thatthere is no operation target.

For example, if the user 106 speaks“Katamen-genko-o-ryomen-de-kopi-shite (Copy one-sided originals on bothsides)” relevant to the operation on the two-sided copy settings screen2300 being displayed, the group-ID determination result {ID: SID00001,ID: SHT00006, ID: SID00000, ID: FNC00001} determined by the group-IDdetermining unit 707 is transmitted. The operation-target determiningunit 807 determines that the group-ID determination result partiallymatches the group ID sets associated with the “1-Sided 2-Sided” button2301 in the operation-target determination information in FIG. 23D. Theprocess then proceeds to S2604, where it is determined whether the“1-Sided 2-Sided” button 2301 is a UI component that can be operatedonly by speech. Since the “1-Sided 2-Sided” button 2301 corresponding tothe group-ID determination result is not included in the screen controlinformation shown in FIG. 23C (but is included in FIG. 23B), the processproceeds to S2608. In S2608, the operation-target determining unit 807determines the “1-Sided 2-Sided” button 2301 corresponding to thegroup-ID determination result as the operation target.

Also, for example, if the user 106 speaks“Hiraki-kata-o-joge-biraki-ni-henko (Change the flipping format tocalendar type (up and down))” relevant to the operation on the two-sidedcopy settings screen 2300 being displayed, the group-ID determinationresult {ID: LYT00023, ID: POS00022, ID: LYT00023, ID: OPR00041}determined by the group-ID determining unit 707 is transmitted. Theoperation-target determining unit 807 determines that the group-IDdetermination result partially matches the group ID sets associated withthe display label 2311 in the operation-target determination informationin FIG. 23D. The process then proceeds to S2604, where it is determinedwhether the display label 2311 is a UI component that can be operatedonly by speech. The display label 2311 corresponding to the group-IDdetermination result is included in FIG. 23C showing the screen controlinformation for UI components that can be operated only by speech.Therefore, the process proceeds to S2605. In S2605, the screen controlinformation (shown in FIGS. 24B and 24C) and the operation-targetdetermination information (shown in FIG. 24D) corresponding to theflipping-format advanced settings screen 2400 relevant to the displaylabel 2311 are acquired. In S2606, the operation-target determining unit807 determines whether any of the group ID sets in the operation-targetdetermination information acquired in S2605 is included in the group-IDdetermination result. In this example, the operation-target determiningunit 807 determines that the group ID set “POS000022+LYT00023” in theoperation-target determination information shown in FIG. 24D is includedin the group-ID determination result {ID: LYT00023, ID: POS00022, ID:LYT00023}. In S2607, the label of the UI component having the group IDset determined to be included in the group-ID determination result inS2606 is acquired from the screen control information (FIG. 24B) andstored. In this example, the label “Calendar Type” of the button 2402 isstored. That is, after determining whether a set value to be displayedin the display label 2311 is included in the speech of the user 106, theoperation-target determining unit 807 stores the set value for updatingthe displayed content of the display label 2311 in the subsequentprocessing. Then in S2608, in this example, the operation-targetdetermining unit 807 determines the display label 2311 as the operationtarget.

In S2610, the operation-target determining unit 807 determines whether apress of any of the hardware keys 201 to 210 on the operation panel 509has been detected through the operation I/F 508. If a press of any ofthe hardware keys 201 to 210 has been detected, the process proceeds toS2613 and otherwise proceeds to S2611.

In S2611, the operation-target determining unit 807 determines whetherthe user 106 has operated any UI component displayed on the LCD touchpanel 200. That is, the operation-target determining unit 807 determineswhether there is a UI component whose touch coordinates on the LCD touchpanel 200, acquired by the operation I/F 508, are within thecorresponding touch coordinate range in the operation-targetdetermination information acquired in S2501. For example, if the touchcoordinates are (X: 500, Y: 450), the operation-target determining unit807 determines, as the operation target, the button 2321 included in thetouch coordinate range in the operation-target determination informationin FIG. 23D (S2612). Also, for example, if the touch coordinates are (X:600, Y: 200), since there is no UI component included in the touchcoordinate range in the operation-target determination information, theoperation-target determining unit 807 determines that there is nooperation target (S2609).

In S2613, the operation-target determining unit 807 determines whetherthe hardware key pressed by the user 106 and detected is included in theoperation-target determination information acquired in S2501. If thehardware key is included in the operation-target determinationinformation, the process proceeds to S2614, where the hardware keypressed and detected is determined as the operation target. If thehardware key is not included in the operation-target determinationinformation, the process proceeds to S2609, where the operation-targetdetermining unit 807 determines that there is no operation target. Forexample, if the user 106 has pressed the hardware key (start key) 207,which is included in the operation-target determination information inFIG. 23D, the operation-target determining unit 807 determines thehardware key 207 as the operation target. For example, if the user 106has pressed the hardware key (or any key of the numeric keypad) 201,which is not included in the operation-target determination informationin FIG. 23D, the operation-target determining unit 807 determines thatthere is no operation target.

The process then returns to S2503 in FIG. 25.

In S2503, it is determined whether, in the operation targetdetermination in S2502 (detailed in FIG. 26), the operation-targetdetermining unit 807 has determined any UI component as the operationtarget. If the operation-target determining unit 807 has determined thatthere is no operation target, the process ends here. If the interactivesession continues at this point, the data transmitting and receivingunit 802 may transmit the information indicating that “there is nooperation target” to the server 102, where, upon receiving thisinformation, the data transmitting and receiving unit 702 synthesizesspeech from text data for prompting the user 106 to operate again.

In S2504, the display unit 806 updates the displayed content of thescreen on the basis of the result of the operation target determinationperformed in S2502 and the screen control information acquired in S2501.

For example, if the “1-Sided 2-Sided” button 2301 is determined as theoperation target in S2502, the display unit 806 updates the screen tohighlight the “1-Sided 2-Sided” button 2301 on the basis of the screencontrol information in FIG. 23B.

For example, if the display label 2311 is determined as the operationtarget in S2502 and “Calendar Type” is stored as the label in S2607, thedisplay unit 806 updates the displayed content of the display label 2311to “Calendar Type”.

In S2505, on the basis of the result of the operation targetdetermination performed in S2502 and the screen control informationacquired in S2501, it is determined whether to execute a job. If any jobis to be executed, the process proceeds to S2507 and otherwise proceedsto S2506. For example, if the “1-Sided 2-Sided” button 2301 isdetermined as the operation target in S2502, since there is no internalprocessing in FIG. 23B, it is determined that no job is to be executed.For example, if the start key 207 is determined as the operation targetin S2502, since “start execution of copy job” appears as internalprocessing in FIG. 23B, it is determined that the job is to be executed.

In S2506, the data transmitting and receiving unit 802 transmits thescreen update notification to the server 102, as in S916 of FIG. 9A. Thescreen update notification may be transmitted only when the interactivesession is in progress.

In S2507, the job is executed on the basis of set job parameters.Specifically, the image forming apparatus 101 executes a series of imageforming operations (e.g., copying, scanning, or printing). For example,when a copy job is to be started while the “1-Sided 2-Sided” button 2301is being highlighted on the screen as illustrated in FIG. 23A, the jobis executed by using set values for the “1-Sided 2-Sided” button 2301 asjob parameters.

In S2508, the data transmitting and receiving unit 802 transmits a jobstate notification to the server 102, as in S946 and S950 describedabove. The job state notification may be transmitted only when theinteractive session is in progress.

(Interactions between System and User)

FIGS. 27A to 27D present a schematic diagram for describing exemplaryinteractions between the user 106 who performs speech operations and theinformation processing system responding thereto, illustrated in thesequence diagram of FIGS. 9A to 9C. FIGS. 27A to 27D illustratecorrespondences between, and flows of, speech examples, which are speechoperations of the user 106, and the corresponding LED statuses, eachpresented by the information processing system to indicate the audioprocessing mode of the audio control apparatus 100. FIGS. 27A to 27Dalso illustrate correspondences between, and flows of, the screensdisplayed on the LCD touch panel 200 of the operation panel 509 of theimage forming apparatus 101 and the audio responses reproduced by theaudio control apparatus 100.

First, the information processing system displays the top screen on theLCD touch panel 200 of the operation panel 509 (S2701). The LED 312 isoff and this indicates that the audio control apparatus 100 is instandby mode. If the user 106 speaks a wake word at this point (S2702),the information processing system starts to accept a speech operationand illuminates the LED 312 to indicate that the audio control apparatus100 is in speech processing mode (S2703). Note that S2702 and S2703 arepresented as examples of S905 and S906, respectively, in the sequencediagram of FIG. 9A. When the user 106 speaks “I want to make a copy”(S2704) after the wake word, the information processing system blinksthe LED 312 to indicate that the audio control apparatus 100 is inresponse processing mode (S2705) and displays the copy top screen(S2706). Then, the information processing system reproduc0es “Copyscreen has been displayed” as an audio response (S2707) and illuminatesthe LED 312 again to prompt the user 106 to speak (S2708). Note thatS2704, S2705, S2706, S2707, and S2708 are presented as examples of S907,S908, S915, S919, and S920, respectively, in the sequence diagram ofFIG. 9A.

When the user 106 speaks “Change the ratio” (S2709), the informationprocessing system blinks the LED 312 (S2710) and displays a ratiosetting screen (S2711). Then, the information processing systemreproduces “You can change the copy ratio” as an audio response (S2712)and illuminates the LED 312 (S2713). Note that S2709, S2710, S2711,S2712, and S2713 are presented as examples of S921, S922, S915, S924,and S925, respectively, in the sequence diagram of FIGS. 9A and 9B.

When the user 106 speaks “I want to copy the A4 sheet onto B5 sheet”(S2714), the information processing system blinks the LED 312 (S2715)and updates the screen to reflect the change in set ratio (S2716). Then,the information processing system reproduces “Copy ratio has been set at86%” as an audio response (S2717) and illuminates the LED 312 (S2718).Note that S2714, S2715, S2716, S2717, and S2718 are presented asexamples of S926, S927, S915, S929, and S925, respectively, in thesequence diagram of FIGS. 9A and 9B.

When the user 106 speaks “Ratio setting has been finished” (S2719), theinformation processing system blinks the LED 312 (S2720) and displaysthe copy top screen reflecting the change in set ratio (S2721). Then,the information processing system reproduces an audio response “You canstart copying” (S2722) and illuminates the LED 312 (S2723). Note thatS2719, S2720, S2721, S2722, and S2723 are presented as examples of S931,S932, S915, S934, and S935, respectively, in the sequence diagram ofFIGS. 9A to 9C.

When the user 106 speaks “Start copying” (S2724), the informationprocessing system blinks the LED 312 (S2725) and starts to execute thecopy job. Upon starting the copy job, the information processing systemdisplays a screen indicating that the copy job is in progress (S2726)and reproduces an initial audio response “Copying will start” (S2727).Upon completion of the copy job, the information processing systemdisplays, for a certain length of time, a screen indicating that thecopy job has been completed, reproduces a final audio response “Copyinghas been completed” (S2728), and turns off the LED 312 to indicate thatthe audio control apparatus 100 has stopped accepting speech (S2729).Note that S2724, S2725, S2726, S2727, S2728, and S2729 are presented asexamples of S936, S937, S944, S949, S954, and S955, respectively, in thesequence diagram of FIG. 9C.

FIGS. 28A to 28E are for explaining an example that can improveoperability in making image forming instructions. Specifically, FIGS.28A to 28E schematically illustrate how the display unit 806 updates thescreen in the example illustrated in FIGS. 23A to 23E and FIGS. 24A to24E.

FIGS. 28A to 28C illustrate how, using the screen control informationshown in FIG. 23B and FIG. 24B, the display unit 806 updates the screenwhen an operation for changing the setting of the page-flipping formatof two-sided copies is performed. Assume that the user 106 wants tochange the set value displayed in the display label 2311 on thetwo-sided copy settings screen 2300. In the touch operation, forexample, after touching the “Set Details” button 2321, the user 106touches the button 2401 or 2402 on the flipping-format advanced settingsscreen 2400 subsequently displayed. The set value displayed in thedisplay label 2311 is not changed until the user 106 presses the OKbutton 2411. Thus, when there is a UI component designed only fordisplaying a set value that can be set on another screen, the example oftouch operation on the two-sided copy settings screen 2300 requires atleast three touches and screen updating associated therewith.

FIGS. 28D and 28E illustrate how, using the screen control informationshown in FIG. 23C, the display unit 806 updates the screen when anoperation for changing the setting of the page-flipping format oftwo-sided copies is performed by speech. For example, assume that theuser 106 speaks “Hiraki-kata-o-joge-biraki-ni-henko (Change the flippingformat to calendar type (up and down))” during display of the two-sidedcopy settings screen 2300. In this case, as illustrated in FIGS. 28D and28E, the set value displayed in the display label 2311 is changed from“Book Type” to “Calendar Type” without transition to the flipping-formatadvanced settings screen 2400 for selecting the page-flipping format oftwo-sided copies. That is, in this example of speech operation, thedisplayed set value can be changed by a single speech of the user 106.It is thus possible to improve operability in making image forminginstructions.

(Conditions for Applying “Set-Value Change Control without ScreenTransition”)

FIG. 29 is a flowchart for generally explaining conditions under whichspecific control is applicable. By this control, a displayed set valueis changed by a single speech of the user 106. Processing represented bythe flowchart of FIG. 29 is executed on the operation screen thatincludes, as illustrated in FIGS. 28D and 28E, a UI component designedonly for displaying a set value that can be set on another screen.Hereinafter, the control described above will be referred to as“set-value change control without screen transition”.

To determine whether the “set-value change control without screentransition” is applicable, it is first determined, in S2901, whether acurrently displayed screen includes a UI component used to transition toa screen at the next level where a set value is changed. For example, ifthe currently displayed screen is the two-sided copy settings screen2300 (see FIG. 23A), which includes the “Set Details” button 2321 fortransition to the flipping-format advanced settings screen 2400 (seeFIG. 24A) at the next level, the process proceeds to S2902, where thenext condition is determined. On the other hand, if the currentlydisplayed screen is the flipping-format advanced settings screen 2400,which does not include a UI component for transition to the settingscreen at the next level, the “set-value change control without screentransition” described above is not applicable (S2907).

To determine whether the “set-value change control without screentransition” is applicable, it is further determined, in S2902, whetherthe currently displayed screen includes, aside from the UI componentdetermined in S2901, a UI component that displays a set value that canbe changed on the screen at the next level. For example, if thecurrently displayed screen is the two-sided copy settings screen 2300(see FIG. 23A), which includes, aside from the “Set Details” button2321, the display label 2311 designed only to display a set value thatcan be changed on the flipping-format advanced settings screen 2400, theprocess proceeds to S2903, where the next condition is determined. Onthe other hand, if the currently displayed screen is the color selectionscreen 2200 (see FIG. 22A), which includes the buttons 2204 and 2205 fortransition to the setting screen (not shown) at the next level but doesnot include any UI component designed only to display a set value thatcan be changed on the setting screen at the next level, the “set-valuechange control without screen transition” is not applicable (S2907).

In S2903, it is determined whether the operation accepted from the user106 is a speech operation. If so, the process proceeds to S2904 wherethe next condition is determined, and otherwise the “set-value changecontrol without screen transition” is not applicable (S2907).

In S2904, it is determined whether speech spoken as a speech operationby the user 106 includes content intended to change the set valuedisplayed in the UI component determined in S2902. Again, this is todetermine whether the “set-value change control without screentransition” is applicable. Then in S2905, it is determined whetherspeech spoken as a speech operation by the user 106 includes contentcorresponding to a set value that can be changed on the setting screenat the next level. Again, this is to determine whether the “set-valuechange control without screen transition” is applicable. For example, ifthe user 106 speaks “Kopi-no-hiraki-kata-o-joge-biraki-ni-henko (Changethe page-flipping format to calendar type (up and down))” during displayof the two-sided copy settings screen 2300 (see FIG. 23A), then the“set-value change control without screen transition” is applicable. Thisis determined using the screen control information and theoperation-target determination information for the corresponding screen(acquired in S2501 and S2605) and the group-ID determination resultgenerated from audio data representing speech of the user 106 (see S2602and S2606). To meet the applicable conditions in S2904 and S2905, thescreen control information and the operation-target determinationinformation are managed in advance by the data management unit 803.

(Remarks)

As described above, in the present embodiment, the screen displayed onthe operation panel 509 of the image forming apparatus 101 can beoperated by speech of the user 106. Particularly in the presentembodiment, when a set value (before change) or setting item name isdisplayed on the current screen and there is another screen (advancedsettings screen) for setting the set value, the set value can be changed(updated) without displaying the advanced settings screen. Then, theupdated set value is eventually displayed. Thus, by speech of the user106, operability in making image forming instructions is improved.

Note that a setting item with a set value that can be changed by speechoperation may be a setting item with a set value that can be changed ona destination screen reached by transition from the currently displayedscreen. In other words, for a setting item with a set value that cannotbe changed on a destination screen (first screen) reached by transitionfrom the currently displayed screen (second screen), no speech-basedoperation is to be accepted. That is, even by speech with the samecontent, the corresponding setting may not be changed depending on thescreen currently displayed. This is to prevent the setting from beingaccidentally changed in response to user's speech that is made on ascreen (third screen) irrelevant to the setting to be changed.

Second Embodiment (Screen Control Information and Operation-TargetDetermination Information)

The first embodiment has described an example in which, for a settingitem with a set value that cannot be changed by touch operation on acurrently displayed screen (second screen) but can be changed on atransition destination screen (first screen), the set value is changedby speech operation. A second embodiment will describe an example inwhich, for a setting item with a set value that cannot be changed bytouch operation on a currently displayed screen (second screen) but canbe partially changed on a transition destination screen (first screen),a procedure for partially changing the setting is performed by speechoperation. The setting item for which the procedure for partiallychanging the setting is performed on the transition destination screen(first screen) is a setting item for which, for example, the procedurefor changing the setting is completed on a screen (third screen) reachedby transition from the first screen. Note that the components used toprovide the second embodiment are basically the same as those of thefirst embodiment, except some features to be described in detail. Thesame components as those of the first embodiment are denoted by the samereference numerals and their detailed description will be omitted.

FIG. 30A illustrates a screen displayed by the display unit 806, andFIG. 30D illustrates a Japanese version of the screen illustrated inFIG. 30A.

FIG. 30A illustrates a print job settings screen 3000 that displaysprint job settings received from the client terminal 103 by the datatransmitting and receiving unit 802. The print job settings screen 3000is used to give an instruction to start printing. The print job settingsscreen 3000 allows the user 106 to check the print job settings andchange the settings as necessary. The print job settings screen 3000includes UI components 3001, 3002, 3003, 3004, 3011, 3012, 3013, 3014,3021, 3022, 3031, and 3032 that can be operated by the user 106. Also,the print job settings screen 3000 presents name information, such as “Nin 1” (the number of pages to be printed on one sheet of paper), whichindicates the name of an item for which an instruction can be given byspeech operation.

The screen control information in FIG. 30B shows that when, for example,the button 3002 labeled “Change” is determined as the operation target,the display unit 806 performs control to enable transition to an “N in1” settings screen 3100 illustrated in FIG. 31A.

The screen control information in FIG. 30C shows that unlike the UIcomponents included in the screen control information in FIG. 30B, thedisplay labels 3011 to 3014 are UI components that cannot be operated bytouch of the user 106 and can be operated only by speech of the user106. For example, the display label 3012 is a UI component fordisplaying a set value that is set on the “N in 1” settings screen 3100illustrated in FIG. 31A. The screen control information in FIG. 30C alsoshows that when the display label 3012 is determined as the operationtarget, the corresponding control to be performed varies depending onthe set value. The data management unit 803 of the present embodimentthus can manage different screen control for each set value set byspeech operation.

FIG. 30C shows exemplary speech operations performed by the user 106 whoviews the screen illustrated in FIG. 30D.

The operation-target determination information in FIG. 30E shows thatwhen the group-ID determination result is {ID: LYT00030, ID: LYT00003,ID: OPR00041} or {ID: LYT00030, ID: LYT00000, ID: OPR00041}, the button3002 is determined as the operation target. For example, assume that theuser 106 speaks “Peji-shuyaku-o-henko (Change the “N in 1” (number ofpages to be printed per sheet))” during display of the print jobsettings screen 3000. In this case, from the group-ID determinationresult {ID: LYT00030, ID: LYT00003, ID: OPR00041} generated by thegroup-ID determining unit 707, the operation-target determining unit 807determines that the button 3002 is the operation target. Then, on thebasis of this determination result and the screen control information,the “N in 1” settings screen 3100 illustrated in FIG. 31A is displayed.

Also, the operation-target determination information in FIG. 30E showsthat when the group-ID determination result is {ID: LYT00030, ID:LYT0003, ID: LYT00010 to LYT00016, ID: OPR00041}, the display label 3012is determined as the operation target. For example, assume that the user106 speaks “Peji-shuyaku-o-tsuinwan-ni-henko (Change the “N in 1”(number of pages to be printed per sheet) to “2 in 1”)” during displayof the print job settings screen 3000. In this case, from the group-IDdetermination result {ID: LYT00030, ID: LYT0003, ID: LYT00011, ID:OPR00041} generated by the group-ID determining unit 707, theoperation-target determining unit 807 determines that the display label3012 is the operation target. The data management unit 803 manages thegroup ID sets in the operation-target determination information suchthat they include group IDs corresponding to UI components of the screen(“N in 1” settings screen 3100) to be subsequently displayed bytransition.

The operation-target determination information in FIG. 30E also showsthat the buttons 3001, 3002, 3003, 3004, 3021, 3022, 3031, and 3032,each having rectangle information representing the touch coordinaterange, are capable of accepting a touch operation on the LCD touch panel200. The hardware key 203, for which the corresponding cell of the touchcoordinate range in FIG. 30E is provided with a diagonal line, is a UIcomponent that is determined as the operation target on the basis of thedetection of a press of the hardware key included in the operation panel509. In FIG. 30E, the touch coordinate range of each of the displaylabels 3011 to 3014 is indicated by “−”. This means that the displaylabels 3011 to 3014 appear on the LCD touch panel 200, but are notdesigned to accept a touch operation.

FIG. 31A illustrates a screen displayed by the display unit 806, andFIG. 31E illustrates a Japanese version of the screen illustrated inFIG. 31A. FIG. 32A illustrates a screen displayed by the display unit806, and FIG. 32C illustrates a Japanese version of the screenillustrated in FIG. 32A. FIG. 32B illustrates another screen displayedby the display unit 806, and FIG. 32D illustrates a Japanese version ofthe screen illustrated in FIG. 32B.

FIG. 31A illustrates the “N in 1” settings screen 3100 displayed by thedisplay unit 806 and configured to allow the user 106 to select thenumber of pages to lay out on one sheet of paper. The “N in 1” settingsscreen 3100 includes UI components 3101, 3102, 3103, 3111, 3121, and3122 that can be operated by the user 106.

FIGS. 31B and 31C show screen control information for the “N in 1”settings screen 3100. FIG. 31D shows operation-target determinationinformation for the “N in 1” settings screen 3100.

FIG. 31D shows exemplary speech operations performed by the user 106 whoviews any of the screens illustrated in FIG. 31E, FIG. 32C, and FIG.32D.

When the “Set Details” button 3111 (FIG. 31A) is pressed, with thebutton 3102 labeled “2 in 1” selected by the user 106, the “N in 1”settings screen 3100 transitions to a “2 in 1” advanced settings screen3200 (FIG. 32A) for selecting the order of pages to lay out. Similarly,when the “Set Details” button 3111 (FIG. 31A) is pressed, with thebutton 3103 labeled “4 in 1” selected by the user 106, the “N in 1”settings screen 3100 transitions to a “4 in 1” advanced settings screen3210 (FIG. 32B) for selecting the order of pages to lay out. Incontrast, when the button 3101 labeled “1 in 1” is selected by the user106, since there is no need to set the order of pages to lay out, the“Set Details” button 3111 is disabled to reject operation from the user106.

(Processing Flow of Device Control Program of Image Forming Apparatus)

FIG. 33 is a flowchart illustrating an outline of screen updateprocessing performed by the device control program 801 of the imageforming apparatus 101. The screen update processing is executed afteraccepting a user's operation. For example, the screen update processingis executed after the data transmitting and receiving unit 802 receivesa group-ID determination result. Alternatively, the screen updateprocessing is executed after the operation I/F 508 detects a touchoperation on the LCD touch panel 200 or detects a press of any of thehardware keys 201 to 210 on the operation panel 509. The relationbetween acceptance of input operation and screen update processing isthe same as that illustrated in the flowchart of FIG. 25. The operationtarget determination is the same as that illustrated in the flowchart ofFIG. 26.

FIG. 33 is a flowchart illustrating details of the screen updateprocessing in S2504, which is part of the processing of the devicecontrol program 801 illustrated in FIG. 25.

Hereinafter, a processing flow performed while the display unit 806 isdisplaying the print job settings screen 3000 in FIG. 30A will bedescribed as an example.

In S3301, the display unit 806 reads the screen control information forthe UI component determined as the operation target in step S2502 (FIG.25). The display unit 806 then determines whether the content of screencontrol information is registered separately for each set value. If thecontent of screen control information is not registered separately foreach set value, the process proceeds to S3302 and otherwise proceeds toS3303. The set value mentioned here is a set value displayed in any ofthe display labels 3011 to 3014 and determined (in S2605 to S2607) to beincluded in the speech of the user 106.

For example, assume that the operation-target determining unit 807determines the display label 3013 as the operation target in S2502. Inthis case, as shown in the screen control information in FIG. 30C, thecontent of screen control information is not registered separately foreach set value. Accordingly, the set value stored in S2607 is controlledto be displayed in the display label 3013. Specifically, if the user 106speaks “Hochikisu-o-settei-nashi-ni-henko (Turn stapling off)”, thedisplay label 3013 is determined as the operation target. At the sametime, the set value “OFF” is stored in S2607 and reflected in thedisplay label 3013.

For example, assume that the operation-target determining unit 807determines the display label 3012 as the operation target in S2502. Inthis case, the screen control information is registered separately foreach set value. Therefore, the screen control is performed in accordancewith the set value stored in S2607. Specifically, if the user 106 speaks“Peji-shuyaku-o-wan′inwan-ni-henko (Change the “N in 1” (number of pagesto be printed per sheet) to “1 in 1”)”, the display label 3012 isdetermined as the operation target and, at the same time, the set value“1 in 1” is stored in S2607. In accordance with the screen controlinformation in FIG. 30C, the set value “1 in 1” is reflected in thedisplay label 3012. If the user 106 speaks“Peji-shuyaku-o-tsuinwan-ni-henko (Change the “N in 1” (number of pagesto be printed per sheet) to “2 in 1”)”, the display label 3012 isdetermined as the operation target and, at the same time, the set value“2 in 1” is stored in S2607. In accordance with the screen controlinformation in FIG. 30C, the display unit 806 enables transition to the“2 in 1” advanced settings screen 3200.

FIGS. 34A to 34E and FIGS. 35A to 35G are for explaining an example thatcan improve operability in making image forming instructions.Specifically, FIGS. 34A to 34E and FIGS. 35A to 35G schematicallyillustrate how the display unit 806 updates the screen in the exampleillustrated in FIGS. 30A to 30E, FIGS. 31A to 31E, and FIGS. 32A to 32D.

FIGS. 34A to 34C illustrate how, using the screen control informationshown in FIG. 30B and FIG. 31B, the display unit 806 updates the screenwhen an operation for changing the setting of “N in 1” is performed.Assume that the user 106 wants to change the set value displayed in thedisplay label 3012 on the print job settings screen 3000. In the touchoperation, for example, after touching the “Change” button 3002, theuser 106 touches the button 3101 on the “N in 1” settings screen 3100subsequently displayed. The set value displayed in the display label3012 is not changed until the user 106 touches the OK button 3122. Thus,when there is a UI component designed only for displaying a set valuethat can be set on another screen, the example of touch operation forthe print job settings screen 3000 requires at least three touches andscreen updating associated therewith.

In this example, the user 106 performs an operation for changing the setvalue to “1 in 1”, which does not require additional setting (i.e.,selecting the order of pages to lay out). Therefore, the operation forchanging the set value is completed by operation on the “N in 1”settings screen 3100 alone.

FIGS. 34D and 34E illustrate how, using the screen control informationshown in FIG. 30C, the display unit 806 updates the screen when anoperation for changing the setting of “N in 1” is performed by speech.For example, assume that the user 106 speaks“Peji-shuyaku-o-wan'inwan-ni-henko (Change the “N in 1” (number of pagesto be printed per sheet) to “1 in 1”)” during display of the print jobsettings screen 3000. In this case, as illustrated in FIGS. 34D and 34E,the set value displayed in the display label 3012 is changed from “2 in1” to “1 in 1” without transition to the “N in 1” settings screen 3100for selecting the set value for “N in 1”. That is, in this example ofspeech operation, the displayed set value can be changed by a singlespeech of the user 106. It is thus possible to improve operability inmaking image forming instructions. Also, when “1 in 1” is selected, noadditional setting is required. This means that the set value to bechanged can be fully determined only by the speech“Peji-shuyaku-o-wan'inwan-ni-henko (Change the “N in 1” (number of pagesto be printed per sheet) to “1 in 1”)” of the user 106. The displayedset value can thus be changed by a single speech of the user 106.

Like FIGS. 34A to 34C, FIGS. 35A to 35D illustrate how, using the screencontrol information shown in FIG. 30B and FIG. 31B, the display unit 806updates the screen when an operation for changing the setting of “N in1” is performed. Assume that the user 106 wants to change the set valuedisplayed in the display label 3012 on the print job settings screen3000. In the touch operation, for example, after touching the “Change”button 3002, the user 106 touches the button 3102 on the “N in 1”settings screen 3100 subsequently displayed. When “2 in 1” is selected,the user 106 needs to select, on the “2 in 1” advanced settings screen3200, the order of pages to lay out as additional setting. Therefore,the user 106 touches the “Set Details” button 3111 as in FIG. 35B. Then,by a touch operation on the “2 in 1” advanced settings screen 3200, theset value representing the order of pages to lay out is changed, and theset value displayed in the display label 3012 on the print job settingsscreen 3000 is eventually changed.

Like FIGS. 34D and 34E, FIGS. 35E to 35G illustrate how, using thescreen control information shown in FIG. 30C, the display unit 806updates the screen when an operation for changing the setting of “N in1” is performed by speech. For example, assume that the user 106 speaks“Peji-shuyaku-o-tsuinwan-ni-henko (Change the “N in 1” (number of pagesto be printed per sheet) to “2 in 1”)” during display of the print jobsettings screen 3000. In this case, where “2 in 1” is specified, theuser 106 needs to select the order of pages to lay out as additionalsetting. This means that the set value to be changed has not been fullydetermined only by the initial speech. In this example, therefore,without transition to the “N in 1” settings screen 3100 for selecting “2in 1” (already specified by speech), the “2 in 1” advanced settingsscreen 3200 is displayed which allows additional setting for “2 in 1”.After the order of pages to lay out is selected by operation on the “2in 1” advanced settings screen 3200, the set value displayed in thedisplay label 3012 on the print job settings screen 3000 is eventuallychanged.

(Remarks)

As described above, in the present embodiment, when a set value isdisplayed on the current screen and there is another screen forselecting the set value, operability in making image forminginstructions is improved by speech of the user 106, as in the firstembodiment described above. Moreover, in the present embodiment, whenthere is still another screen (advanced settings screen) for settingdetails of the set value selected on the screen described above, screencontrol that enables direct transition to the advanced settings screenis performed. This improves operability in making image forminginstructions. In other words, in the present embodiment, a speechoperation can eliminate the need for displaying some of the screens usedin a setting change procedure.

As described above, when there is a UI component designed only fordisplaying a set value that can be set on another screen, a settingchange operation may be either completed on this screen or may requirean additional operation for changing advanced settings on still anotherscreen. By performing different screen control for each of these cases,operability in making image forming instructions is improved.

Third Embodiment

The first and second embodiments have described techniques in which, byspeech operation, a set value can be changed without displaying some orall of setting procedure screens displayed for touch operation. A thirdembodiment will describe an example which varies a setting procedurescreen depending on whether a setting change instruction accepted iseither speech or touch-based.

(System Control Sequence)

FIG. 36 is a sequence diagram illustrating interactions between theapparatuses included in the information processing system illustrated inFIG. 1 and also between the control programs of the apparatusesillustrated in FIG. 6, FIG. 7A, and FIG. 8. FIG. 36 illustrates anotherexample of S921 to S930 in FIGS. 9A and 9B. In FIG. 36, steps equivalentto S905 to S920 in FIG. 9A and S936 to S956 in FIG. 9C are omitted. Thedescription of components, which are the same as those illustrated inFIG. 1 to FIG. 8, will also be omitted.

In S3621, as in S921, the user 106 instructs the audio control apparatus100 to call a setting screen. For example, assume that a setting screenfor adding an e-mail subject and message to be sent is displayed over ascreen for executing the function of e-mailing an image of a scannedoriginal. At this point, as an instruction to call a setting screen, theuser 106 speaks, for example, “I want to enter the message” with theintention of adding any character string as the e-mail body.

In S3622, the same operation as that in S922 is performed.

In S3623, the speech operation processing similar to that performed inS923 is performed. In S3623, for example, in accordance with theinstruction (sent in S3621) to call a setting screen, the display unit806 updates the screen to display a screen for entering an e-mailmessage. At the same time, as a screen update notification, the datatransmitting and receiving unit 802 transmits text data “message entryscreen displayed” to the server 102. Upon receiving this text data, thedata transmitting and receiving unit 702 of the audio-data conversioncontrol program 701 remembers that the system is ready to receive inputof any text.

In S3624, as in S924, the audio reproducing unit 605 reproducessynthesized speech data. For example, the audio reproducing unit 605reproduces the synthesized speech data representing “You can type byvoice. Please read out the message you want to type in” through thespeaker 310.

In S3625, the same operation as that in S925 is performed.

In S3626, the user 106 speaks, toward the audio control apparatus 100,any words or sentences to enter, so as to instruct the audio controlapparatus 100 to receive input of any characters. The speech of the user106 is acquired by the speech acquiring unit 604 and generated as audiodata. After the elapse of a predetermined blank period, the speech-enddetermining unit 608 determines that the speech has ended.

In S3627, the same operation as that in S927 is performed.

After the audio data is generated as an instruction to receive input ofany characters in S3626, the data transmitting and receiving unit 602transmits the generated audio data to the server 102 in S3628.

In S3629, the speech recognition unit 705 of the audio-data conversioncontrol program 701 performs speech recognition on the audio datareceived by the data transmitting and receiving unit 702. By the speechrecognition, for example, text date is generated from the speech“Sakihodo-no-shiryo-o-sofu-itashimasu-node-go-kakunin-kudasai (Pleasefind the attached materials mentioned earlier)” spoken by the user 106.In S3630, the data transmitting and receiving unit 702 of the audio-dataconversion control program 701 transmits the text data, which is thespeech recognition result, to the image forming apparatus 101.

In S3631, the content of the text data or speech recognition resultreceived by the data transmitting and receiving unit 802 of the devicecontrol program 801 is reflected in the screen displayed by the displayunit 806. For example, instead of touching a keyboard displayed on theLCD touch panel 200 or pressing the numeric keypad 201 to enter anycharacter string, the user 106 speaks to directly enter the content ofthe speech as a character string.

In S3632, the data transmitting and receiving unit 802 transmits, to theserver 102, a screen update notification indicating that displayedcontent of the screen has been updated. For example, as a screen updatenotification, the display unit 806 transmits text data “character inputcompleted”.

In S3633, the speech synthesis unit 708 of the audio-data conversioncontrol program 701 performs speech synthesis on the screen updatenotification received by the data transmitting and receiving unit 702.Specifically, the speech synthesis unit 708 synthesizes speech frompredetermined text data corresponding to the screen update notification.For example, if the screen update notification is “message inputcompleted”, the speech synthesis unit 708 synthesizes the speech “Pleaseconfirm the message entered” from the corresponding text data.

In S3634, the audio data (synthesized speech data) generated throughspeech synthesis in S3633 by the speech synthesis unit 708 istransmitted by the data transmitting and receiving unit 702 to the audiocontrol apparatus 100. The synthesized speech data is received by thedata transmitting and receiving unit 602 of the audio control program601.

In S3635, the audio reproducing unit 605 reproduces the synthesizedspeech data received in S3634. For example, the synthesized speech data“Please confirm the message entered” generated in S3633 is reproducedthrough the speaker 310.

In S3636, after the synthesized speech data is reproduced, the displayunit 606 of the audio control program 601 illuminates the LED 312 againto indicate that the audio control apparatus 100 is in the speechprocessing mode. At the same time, the speech acquiring unit 604 startsprocessing again. (Screen Control Information and Operation-TargetDetermination Information)

FIG. 37A illustrates a screen displayed by the display unit 806, andFIG. 37E illustrates a Japanese version of the screen illustrated inFIG. 37A.

FIG. 37A illustrates an e-mail message setting screen 3700 displayed bythe display unit 806 for the user 106 to set a subject and message fore-mail transmission. The e-mail message setting screen 3700 includes UIcomponents 3701, 3702, 3711, and 3712 that can be operated by the user106.

The screen control information in FIG. 37B shows that when, for example,the button 3711 labeled “Cancel Settings” is determined as the operationtarget, the display unit 806 performs control so as to enable transitionto a send top screen (not shown), which is a top screen for the functionof e-mailing an image of a scanned original. The screen controlinformation in FIG. 37B also shows that when, for example, the button3702 labeled “Message” is determined as the operation target, the screencontrol to be performed varies depending on whether the type of inputoperation is touch or speech. The data management unit 803 of thepresent embodiment is thus capable of managing, as screen controlinformation, the content of screen control that varies depending on thetype of input operation. The screen control information in FIG. 37Bshows that if, for example, the button 3702 is determined as theoperation target in response to touch operation, the screen transitionsto a keyboard input screen 3800 (see FIG. 38A), whereas if the button3702 is determined as the operation target in response to speechoperation, the screen transitions to a speech input screen 3810 (seeFIG. 38B).

The screen control information in FIG. 37C is empty. This indicates thatthe e-mail message setting screen 3700 does not include a UI componentthat can be operated only by speech.

The operation-target determination information in FIG. 37D shows thatwhen the group-ID determination result is {ID: SND00062}, {ID: SND00062,ID: OPR00040}, or {ID: SND00062, ID: OPR00050}, the button 3702 isdetermined as the operation target. For example, assume that the user106 speaks “Meru-honbun-o-settei (Set the e-mail message)” duringdisplay of the e-mail message setting screen 3700. In this case, fromthe group-ID determination result {ID: SND00062, ID: OPR00040} generatedby the group-ID determining unit 707, the operation-target determiningunit 807 determines that the button 3702 is the operation target.

The operation-target determination information in FIG. 37D also showsthat the buttons 3701, 3702, 3711, and 3712, each having rectangleinformation representing the touch coordinate range, are capable ofaccepting the touch operation on the LCD touch panel 200. For example,when touch coordinates are (X: 200, Y: 250), the operation-targetdetermining unit 807 determines the button 3702 included in the touchcoordinate range in the operation-target determination information asthe operation target. The hardware keys 203 and 207, for which thecorresponding cells of the touch coordinate ranges in FIG. 37D are eachprovided with a diagonal line, are UI components that are eachdetermined as the operation target on the basis of the detection of apress of the hardware key included in the operation panel 509.

FIG. 37D shows exemplary speech operations performed by the user 106 whoviews any of the screens illustrated in FIG. 37E, FIG. 38C, and FIG.38D.

FIG. 38A illustrates a screen displayed by the display unit 806, andFIG. 38C illustrates a Japanese version of the screen illustrated inFIG. 38A. FIG. 38B illustrates another screen displayed by the displayunit 806, and FIG. 38D illustrates a Japanese version of the screenillustrated in FIG. 38B.

FIG. 38A illustrates the keyboard input screen 3800 described above. Thekeyboard input screen 3800 includes various keys 3802 of a softwarekeyboard, and a region 3801 that displays any character string set bythe user 106. By touching the various keys 3802 of the software keyboardon the keyboard input screen 3800, the user 106 can enter any characterstring that is to be set as an e-mail message.

FIG. 38B illustrates the speech input screen 3810 described above. Thespeech input screen 3810 includes a region 3811 that displays anycharacter string set by the user 106. By speaking toward the speechinput screen 3810, the user 106 can enter the speech in the form of anycharacter string that is to be set as an e-mail message. The speechinput screen 3810 does not include the various keys 3802 of the softwarekeyboard, which is not required for speech operation. This can make thesize of the region 3811 for displaying any character string set by theuser 106 greater than that of the region 3801. That is, the region 3811can display more character strings set by the user 106.

(Processing Flow of Device Control Program of Image Forming Apparatus)

FIG. 39 is a flowchart illustrating an outline of screen updateprocessing performed by the device control program 801 of the imageforming apparatus 101. The screen update processing is executed afteraccepting a user's operation. For example, the screen update processingis executed after the data transmitting and receiving unit 802 receivesa group-ID determination result. Alternatively, the screen updateprocessing is executed after the operation I/F 508 detects a touchoperation on the LCD touch panel 200 or a press of any of the hardwarekeys 201 to 210 on the operation panel 509. The relation betweenacceptance of input operation and screen update processing is the sameas that illustrated in the flowchart of FIG. 25. The operation targetdetermination is the same as that illustrated in the flowchart of FIG.26.

Hereinafter, a processing flow performed while the display unit 806 isdisplaying the e-mail message setting screen 3700 in FIG. 37A will bedescribed as an example.

In S3901, the display unit 806 reads the screen control information forthe UI component determined as the operation target in step S2502 (FIG.25). The display unit 806 then determines whether the content of screencontrol is registered separately for touch operation and speechoperation. If the content of screen control is not registered separatelyfor touch operation and speech operation, the process proceeds to S3903and otherwise proceeds to S3902. For example, if the operation-targetdetermining unit 807 determines the button 3711 as the operation targetin S2502, the content of screen control is not registered separately fortouch operation and speech operation in the screen control information.Therefore, in S3903, the display unit 806 performs control, on the basisof the content of screen control, so as to enable transition to the sendtop screen (not shown). If, for example, the operation-targetdetermining unit 807 determines the button 3702 as the operation targetin S2502, the content of screen control is registered separately fortouch operation and speech operation in the screen control information.The process thus proceeds to S3902.

In S3902, as in S2601 (FIG. 26), the type of input operation isdetermined. If the type of input operation is speech, that is, if, as inFIG. 9A, the user 106 speaks after instructing to start a speechoperation in S905 and the data transmitting and receiving unit 802receives a group-ID determination result from the server 102, then theprocess proceeds to S3905. If the type of input operation is a touchoperation or a press of a hardware key, the process proceeds to S3904.

In S3904, the display unit 806 reads the content of screen control fortouch operation in the corresponding screen control information andupdates the screen on the basis of the read content. For example, if theuser 106 touches the button 3702, the display unit 806 performs controlto enable transition to the keyboard input screen 3800.

In S3905, the display unit 806 reads the content of screen control forspeech operation in the corresponding screen control information andupdates the screen on the basis of the read content. For example, if theuser 106 speaks “Meru-honbun-o-settei (Set the mail message)” and thebutton 3702 is determined as the operation target, the display unit 806performs control to enable transition to the speech input screen 3810.(Remarks)

As described above, in the present embodiment, when the user 106 entersany character string by speech during a speech-based interactivesession, a screen is displayed, which excludes keys unnecessary forspeech operation and provides a large region for displaying a characterstring entered (see FIGS. 40C and 40D). This improves operability inmaking image forming instructions in speech operation. When the user 106enters any character string by traditional touch operation, a screenhaving a software keyboard is displayed (see FIGS. 40A and 40B). By thusdisplaying a different screen depending on the type of input operationof the user 106, operability in making image forming instructions isimproved.

Other Embodiments

The present disclosure may be applied either to a system including aplurality of devices, or to an apparatus including a single device. Inthe embodiments described above, the audio control program 601, thedevice control program 801, and the audio-data conversion controlprogram 701 are executed by the audio control apparatus 100, the imageforming apparatus 101, and the server 102, respectively. Alternatively,all the control programs described above may be executed by the imageforming apparatus 101 that includes a microphone. The audio controlprogram 601 and the device control program 801 may be executed by theimage forming apparatus 101, with the server 102 alone separated.

The present disclosure is not limited to the embodiments describedabove. Various modifications (including organic combinations of theembodiments) may be made to the present disclosure on the basis of thespirit of the present disclosure, and such modifications should not beexcluded from the scope of the present disclosure. That is, allcombinations of the aforementioned embodiments and their modificationsare included in the present disclosure.

In the embodiments described above, some of the operation screensprovided by the image forming apparatus 101 have been presented, andoperations performed on the screens have been explained. However, screenoperations based on natural language are not limited to those describedabove, and may include operations on any screens displayed by the imageforming apparatus 101. The user 106 may instruct, for example, to changethe setting, switch the screen, and execute the processing by speechoperation.

Embodiment(s) of the present disclosure can also be realized by acomputer of a system or apparatus that reads out and executes computerexecutable instructions (e.g., one or more programs) recorded on astorage medium (which may also be referred to more fully as a‘non-transitory computer-readable storage medium’) to perform thefunctions of one or more of the above-described embodiment(s) and/orthat includes one or more circuits (e.g., application specificintegrated circuit (ASIC)) for performing the functions of one or moreof the above-described embodiment(s), and by a method performed by thecomputer of the system or apparatus by, for example, reading out andexecuting the computer executable instructions from the storage mediumto perform the functions of one or more of the above-describedembodiment(s) and/or controlling the one or more circuits to perform thefunctions of one or more of the above-described embodiment(s). Thecomputer may comprise one or more processors (e.g., central processingunit (CPU), micro processing unit (MPU)) and may include a network ofseparate computers or separate processors to read out and execute thecomputer executable instructions. The computer executable instructionsmay be provided to the computer, for example, from a network or thestorage medium. The storage medium may include, for example, one or moreof a hard disk, a random-access memory (RAM), a read only memory (ROM),a storage of distributed computing systems, an optical disk (such as acompact disc (CD), digital versatile disc (DVD), or Blu-ray Disc(BD)TM), a flash memory device, a memory card, and the like.

While exemplary embodiments have been described, it is to be understoodthat the disclosure is not limited to the disclosed exemplaryembodiments. The scope of the following claims is to be accorded thebroadest interpretation so as to encompass all such modifications andequivalent structures and functions.

What is claimed is:
 1. An information processing system comprising: amicrophone configured to acquire sound; a display device configured todisplay information; and one or more controllers configured to: causethe display device to display a setting screen on which names of aplurality of setting items are arranged; and acquire word informationincluding a combination of a name of one setting item of the pluralityof setting items and a word corresponding to a parameter of the onesetting item based on speech acquired through the microphone duringdisplay of the setting screen, and change the set value of the onesetting item on the basis of the word information, wherein even when theword information including the combination of the name of the onesetting item of the plurality of setting items and the wordcorresponding to the parameter of the one setting item is acquired onthe basis of speech acquired through the microphone during display ofanother screen on which the one setting item is not arranged, the setvalue of the one setting item is not changed.
 2. The informationprocessing system according to claim 1, wherein the setting screenincludes name information of the one setting item.
 3. The informationprocessing system according to claim 1, wherein the word information isa date generated using a learning model trained on the basis of teacherdata containing pairs of text information and audio information.
 4. Theinformation processing system according to claim 1, further comprisingan apparatus including the microphone, the display device, and anacquiring unit configured to acquire the word information.
 5. Theinformation processing system according to claim 1, further comprising:an apparatus including the microphone and the display device; andanother apparatus including an acquiring unit configured to acquire theword information.
 6. The information processing system according toclaim 1, further comprising: an apparatus including the microphone; andanother apparatus including the display device and an acquiring unitconfigured to acquire the word information.
 7. The informationprocessing system according to claim 1, wherein the setting screenincludes information showing a set value not yet changed.
 8. Theinformation processing system according to claim 1, wherein the one ormore controllers is further configured to cause the display device todisplay a changed screen in response to accepting an instruction tochange the set value of the one setting item, the changed screenincluding information showing a changed set value of the one settingitem.
 9. The information processing system according to claim 1, whereinthe setting screen includes a selection object for transition to afurther screen, and the further screen is a screen on which a pluralityof candidates for a parameter settable for the one setting item.
 10. Theinformation processing system according to claim 1, wherein the one ormore controllers is further configured to acquire word informationincluding a name of the one setting item of the plurality of settingitems and not including the word corresponding to the parameter of theone setting item based on speech acquired through the microphone duringdisplay of the setting screen, and cause transition to the furtherscreen, and the further screen is a screen on which a plurality ofcandidates for a parameter settable for the one setting item.
 11. Theinformation processing system according to claim 1, wherein theinformation processing system has a copy function.
 12. The informationprocessing system according to claim 1, wherein the informationprocessing system has a scan function.
 13. The information processingsystem according to claim 1, wherein the information processing systemhas a print function.
 14. The information processing system according toclaim 1, wherein the information processing system has a FAX function.15. The information processing system according to claim 1, wherein theinformation processing system has an E-mail function.
 16. An informationprocessing apparatus comprising: a microphone configured to acquiresound; a display device configured to display information; one or morecontrollers configured to: cause the display device to display a screenon which a instruction is receivable and names of a plurality of settingitems are arranged; and acquire word information including a combinationof a name of one setting item of the plurality of setting items and aword corresponding to a parameter of the one setting item based onspeech acquired through the microphone during display of the screen, andchange the set value of the one setting item on the basis of the wordinformation, wherein even when the word information including thecombination of the name of the one setting item of the plurality ofsetting items and the word corresponding to the parameter of the onesetting item is acquired on the basis of speech acquired through themicrophone during display of another screen on which the one settingitem is not arranged, the set value of the one setting item is notchanged.
 17. The information processing apparatus according to claim 16,wherein the information processing apparatus has a scan function. 18.The information processing apparatus according to claim 16, wherein theinformation processing apparatus has a print function.
 19. Theinformation processing apparatus according to claim 16, wherein theinformation processing apparatus has a FAX function.
 20. The informationprocessing system according to claim 16, wherein the informationprocessing apparatus has an E-mail function.